Review Article | Open Access
Patricia O. Towle, Patricia A. Patrick, Tamique Ridgard, Sofia Pham, Jaime Marrus, "Is Earlier Better? The Relationship between Age When Starting Early Intervention and Outcomes for Children with Autism Spectrum Disorder: A Selective Review", Autism Research and Treatment, vol. 2020, Article ID 7605876, 17 pages, 2020. https://doi.org/10.1155/2020/7605876
Is Earlier Better? The Relationship between Age When Starting Early Intervention and Outcomes for Children with Autism Spectrum Disorder: A Selective Review
Although the conventional wisdom is that “earlier is better” when it comes to intervention for children with ASD, it is not clear what evidence exists to support this notion. This review examined a group of studies that addressed outcomes for young children with ASD who started early intervention at a range of ages. The review was selective by including only papers that examined the age of initiation of treatment as well as baseline cognitive, language, or adaptive level and, in addition, employed a method to control for the covariance between early ability level and age of beginning intervention. Fourteen studies were identified and then compared on methods and outcomes. The support for “earlier is better” was mixed, but it was clear that complex relationships among predictor variables need to be explored in order to understand the role of age of starting early intervention for later outcomes.
Autism spectrum disorder (ASD) is a neurobehavioral disorder that significantly affects social interaction, communication, and interests. The Centers for Disease Control and Prevention recently reported an estimate that 16.8 per 1,000, or one in 59, children aged 8 years have ASD . One of the most important recent research and clinical advances in the field has been the ability to diagnose ASD at increasingly younger ages; the current consensus is that most children with ASD can be diagnosed by 24 months [2–4]. Early diagnosis is now strongly encouraged due to the mounting evidence for the positive effects of early intervention [5–7], which is seen as one of the keys to mitigating the lifelong effects of ASD and associated costs [8, 9]. A corollary consensus is that the earlier a child starts in an intervention program, the better the outcome. However, it is not clear what direct evidence exists demonstrating this to be the case [10, 11].
There are several strong and interrelated rationales for the assertion that earlier is better when it comes to intervention. The first relates to early neuroplasticity, with the birth-to-three period considered a peak neuroplastic phase due to the rate of synaptic formation . The human brain is not functionally mature at birth, but requires extensive interaction with the environment (i.e., experience) for elaborated synaptic connections and cortical specialization in combination with genetically programmed neuromaturation [13, 14]. Early intervention can be seen as highly specialized experience that may shape and even correct patterns being formed during the birth-to-three period.
The concept of early neuroplasticity is further elaborated by the notion of critical or sensitive periods—times during which the brain is primed for specific areas of learning via experience combined with organismic development; an associated feature is that the window incrementally closes up to a certain point. There is now a substantial body of evidence, generated through the Baby Siblings Research Consortium, showing that ASD features in infancy are emerging at the same time that typical early social and communication skill sets are building and consolidating [15–17]. The earliest observable ASD symptoms usually become evident from 12 to 18 months of age. Yet it is highly likely that the basic developmental processes have derailed even earlier—at a prodromal stage—and these processes are both resulting in observable symptoms and interfering with further typical development .
There are now several models of early autism etiology such as the “social motivation” hypothesis [19–21], as well as those based on broader disturbances in attention, biologic motion, and sensory perception. Each of these has accumulating imaging and psychophysical evidence of associated neurologic differences between controls and infants who are later diagnosed with ASD [22, 23]. Yet each model incorporates the concept of a developmental cascade, wherein disturbances of one phase of development significantly interfere with the successful acquisition of those subsequent, mirrored by early neurologic differences that interfere with further cortical specialization and typical skill development [24, 25].
Considering these lines of evidence and theory, a strong argument can be made for intervention for children with ASD to take place as close to early critical periods as possible and certainly when the first autism symptoms appear. It follows that the more time that goes by, the harder it will be to remediate neural circuitry and behavioral expression for more optimal behavioral patterns. It remains unclear, however, to what extent the specific question of the relative effects of earlier intervention has been asked and answered definitively [11, 26, 27].
This paper examines the extant literature on the role that age of starting early intervention plays in the varying outcomes for children with ASD. Studies were identified that included age of starting intervention as a predictor of later capability levels of children diagnosed early with ASD. However, two lines of research informed a selective rather than a broader review. The first is that there is clear evidence that one of the strongest early predictors of outcome in ASD is baseline cognitive functioning [28, 29]. Other significant predictors have likewise reflected early skill levels, such as early language , baseline play , and social interaction capabilities  as key determinants in the long-term outcomes for children with ASD. Therefore, to be considered in this review, along with age of starting early intervention, the study needed to (1) also include as a predictor at least baseline cognitive or language level and (2) determine other predictors’ covariance with age of starting intervention, and in addition, employ a statistical or design method for controlling for shared variance between age of entry into EI and baseline ability level in relation to outcome variables.
These criteria were considered essential because research has shown that there are important child features that increase the likelihood of earlier diagnosis (and thus start of intervention), such as presence of obvious repetitive behaviors [33, 34], greater delays associated with eventual intellectual disability , and more severe communication and social delays . As mentioned, these are features that will have a strong influence on later outcomes and may override the individual effect of starting age. Several environmental variables also have been shown to influence age of diagnosis, including parental education level , geographic region reflecting the density of professional resources, parental attribution of delays, degree of previous interactions with health systems , and cohort effects [33, 35].
These influences make it possible that the younger vs. older children (when beginning EI) in a study may not be similar in terms of strongly influential features such as baseline cognitive level or well-developed parental advocacy. The methods to address these possible confounds in the papers reviewed were anticipated to be a multiple regression design or other strategies that controlled for shared variance among predictors (e.g., ANCOVA).
After a systematic search, the included studies were characterized in terms of research design, participant features, follow-up period, predictor variables, and data analytic methods. Results across studies were synthesized and outcomes interpreted with a focus on the role played by age of entering early intervention.
2. Method: Literature Search and Criteria for Inclusion
The electronic databases of PubMed, PsychINFO, Education Resources Information Center (ERIC), and Google Scholar were searched from 1995 to 2018, using search terms that included various combinations of “infants,” “young children,” “at-risk,” “autism,” “autism spectrum disorder,” “age,” “early intervention,” “prediction,” “longitudinal,” and “outcomes.” Gray literature was not searched. Hand searches also were completed of reference lists from included articles as well as those from four reviews and three meta-analyses. Criteria for inclusion were that the study was published in a peer-reviewed journal, was available in English, used either DSM-IV, DSM-IV-TR, or DSM-5 criteria for autism or ASD diagnosis, and included age of starting early intervention as one focus of inquiry along with other key predictors. As explained in the Introduction, studies for detailed review also needed to inquire past simple group comparisons or zero-order correlations from predictors to outcome, include several other predictor variables besides initial age, and use statistical techniques for controlling for their shared variance.
3.1. Study Selection
The PRISMA flow diagram for the process of selecting final studies for inclusion is shown in Figure 1. After an initial screen to determine that only children with ASD were involved, a treatment component was applied and outcome of treatment was the focus, and age of entry into intervention was included, 1,244 were excluded. Two raters (authors PT and PP) then independently judged whether the abstract should be chosen for a full read. These raters achieved moderate agreement at this stage of study selection (K = 0.64; 95% CI: 0.49, 0.79). Disagreements were resolved through discussion, and 130 articles were read thoroughly and independently by the same two raters; strong agreement was achieved at this final stage (K = 0.89; 95% CI: 0.74, 1.0), with the result of 14 articles to be included for the review.
3.2. Systematic Review of Studies
3.2.1. Range of Studies and Features
Of the 14 included studies, five were from the United States, two were from Canada, four from Australia, and one each from Israel, Spain, and Norway. Five studies looked exclusively at infants and toddlers, while five spanned the birth-to-three and preschool age (3–5 years), and four studies included preschoolers as well as children up to 7 years of age.
Table 1 shows the following for the 14 studies: research design, sample size, age of participants, length of the follow-up period, and measurements used. There were 12 intervention studies and two longitudinal studies. Sample sizes ranged from 24 to 332. The follow-up period after intervention ranged from 12 weeks to 17 years.
1Prediction study conducted after original intervention study was published. 2Prediction study conducted after additional follow-up period. 3Prediction analyses conducted concurrently with treatment effectiveness and presented in one publication. 4Longitudinal study; RCT = randomized controlled trial; TAU = treatment as usual; tx = treatment; DAS = Differential Ability Scales; SPA = Structured Play Assessment; MSEL = Mullen Scales of Early Learning; WPPSI = Wechsler Preschool and Primary Scale of Intelligence; LIPS = Leiter International Performance Scale; ADOS = Autism Diagnostic Observation Schedule; ADI-R = Autism Diagnostic Interview-Revised; VABS = Vineland Adaptive Behavior Scale; PPVT-III = Peabody Picture Vocabulary Test, 3rd Edition; E-LAP = The Early Learning Accomplishment Profile; LAP-D = The Learning Accomplishment Profile-Diagnostic; BED = Best Estimate Diagnosis; ABC = Autism Behavior Checklist; CARS = Childhood Autism Rating Scale; SCQ = Social Communication Questionnaire; EVT = Expressive Vocabulary Test; MCDI = MacArthur–Bates Communicative Development Inventory; ESCS = Early Social Communication Scales; PCX = Parent-Child Interaction; WSQ = Wing Subtypes Questionnaire; DP-3 = Developmental Profile-3; RBS-R = Repetitive Behavior Scale-Revised; LENA = Language ENvironment Analysis; M-COSMIC = Modified Classroom Observation Schedule to Measure Intentional Communication.
In addition to child age at initiation of treatment, all of the studies assessed cognitive functioning, 12 assessed adaptive behavior skills, 13 measured autism severity, nine assessed language skills, and seven measured some dimension of early social functioning. For cognitive measurement, a common instrument was the Mullen Scales of Early Learning (MSEL) . Other measurements used were the following norm-referenced instruments: the Differential Ability Scales (DAS) [38, 39] and the Wechsler Preschool and Primary Scale of Intelligence (WPPSI) . The Vineland Adaptive Behavior Scale (VABS) [41, 42], a norm-referenced parent interview measure, was used in most of the studies to measure adaptive behavior. To assess autism severity, most studies used the Autism Diagnostic Observation Scale (ADOS) . Other autism severity measures were the Childhood Autism Rating Scales (CARS)  and the parent-report Social Communication Questionnaire (SCQ) . Measurement of early social and linguistic skills was the most variable, using language scales and a number of laboratory tasks and observational measures for gestures, joint attention, imitation, social reciprocity, and parent-child interaction (see Table 1).
3.2.2. How Studies Controlled for Time 1 Ability Level Covariance with Age
Table 2 is organized around the four general research designs that were used across studies. It can be seen that this design aspect tended to determine the approach for investigating and controlling for the intercorrelations between age of starting early intervention and Time 1 ability levels. Studies that identify either younger- vs. older-starting groups at the beginning or higher and lower outcome groups at the end of intervention or after a follow-up period were more likely to directly examine early developmental levels associated with age.
3.2.3. Results of Studies (Table 3)
Twelve of the 14 studies demonstrated that age of starting early intervention contributed significantly to outcome of children with ASD. The following describes the studies and their findings in this regard.
Studies that demonstrated no significant beneficial predictive effect of age of starting intervention or were equivocal are as follows:(1)Hedvall et al. (2015) : this study examined children who had been treated with ABA-based early intervention in community programs. They identified a group who had gained the most (GM) (N = 30) and those who had lost the most (LM) (N = 23) over two years of intervention, based on a high and low 15th percentile cutoff on outcome VABS scores. Those who had lost the most had been referred at an earlier age and examination of T1 attributes suggested that these children were more impaired initially. In a logistic regression of these predictors, only Time 1 cognitive level contributed independent variance to the outcome variable of adaptive behavior scores.(2)Tiura et al. (2017) : this was a trajectory analysis study, and tracking patterns of change were associated with ABA intervention for 35 children aged 2–4 years at the beginning of the treatment. The measure used to track growth was the Developmental Profile-3rd Edition (DP-3) , measuring its subdomains individually; these were administered every six months over three years. Multilevel modeling showed a linear growth model fit better than quadratic or cubic across the domains. In terms of predictors, the authors addressed the prediction/strength of association between all Time 1 variables first. However, these were not developmental quotients (which are ratios) but absolute age scores, and as a result, older children had higher scores. In their subsequent longitudinal prediction analyses, age of starting early intervention was not a significant predictor for outcome scores, whereas Time 1 ability levels such as cognitive and adaptive scores were. A second question was that of predicting growth rates. In this case, there was a nonsignificant trend for earlier age of entry to EI to predict faster growth rates.(3)Eapen et al. (2016) : this study examined predictors of outcome in preschool children (N = 49) aged 3 to 5years when starting approximately 10 months of Early Start Denver Model (ESDM) group intervention. A linear regression controlled for baseline IQ, autism severity, and adaptive behavior. Initial age did account for a significant amount of the variance for autism severity as measured by parent-rated SCQ. On the other hand, this finding did not apply to outcomes that are usually considered important for demonstrating the effectiveness of intervention, that is, examiner-measured IQ, autism severity, and adaptive behavior. It is not evident why a parent-report measure is the only measure supporting the thesis.
Studies demonstrating the significant beneficial predictive effect of age of starting intervention using a variety of analytic strategies are as follows:(1)Kasari et al. (2012) : the original study [50, 51] randomly assigned 58 preschoolers (aged 3 to 4 years) into two treatment groups and one nontreatment control group. For their later 2012 prediction study, reported on here, 40 of these children were re-examined five years later. The authors divided the children into those that reached a functional language level (approximately 2 years, 6-month level, N = 32) and those that did not (N = 8). A second grouping was those achieving a cognitive level at the same age equivalent (2 years, 6 months) versus not. Two other outcome targets were cognitive level and expressive vocabulary level (Expressive Vocabulary Test (EVT)) . When a hierarchical multiple regression was then performed for the cognitive outcome, only baseline functional play level predicted the membership of the outcome group. However, for level of spoken vocabulary at 8 to 9 years, starting age, joint attention, play level, and treatment group assignment all contributed significantly. “On average, the children gained a standard score of 1.1 (SE = 0.3) in spoken vocabulary ability per month that they entered the treatment earlier.”(2)Smith et al. (2015) : an intervention study of the Lovaas-based Young Autism Program was conducted as carried out in the community with a high level of supervision. Seventy-one children with ASD, aged 20 to 59 months, received two years of preschool-based intervention. The authors conducted sequential multiple regressions for each outcome measure separately for Year 1 and Year 2. The finding related to age of starting the intervention was as follows: controlling for all other covariates (IQ, adaptive skills, and autism severity), lower age at intake predicted higher outcome for MSEL and, marginally, for VABS and ADOS scores. An interaction effect was found for Age × Year of Progress, which is reported as follows.(3)Virues-Ortega et al. (2013) : longitudinal multilevel modeling was used to identify predictors of outcomes of Intensive Behavioral Intervention (IBI) based on the Early Learning Accomplishment Profile (E-LAP) [54, 55] and the Learning Accomplishment Profile-Diagnostic, 3 Edition (LAP-D) . Twenty-four children diagnosed with ASD (M = 50.0 months, SD = 28.3) participated in IBI over two years on average and were assessed using the ELAP every 6 months. A trajectory analysis demonstrated that the best solution for this relatively small sample was for two different groups, one higher performing over time and the other lower performing. The lower-performing children started with lower scores. In contrast, the higher-performing children, who started with higher scores, were less likely to decelerate and reach a ceiling in age-level measurement during the intervention period. Three child-associated variables were explored in two-predictor models. Age child started intervention best improved the fit for gross motor function, receptive language, self-care, and social behavior, while preintervention functioning level was the second most efficient predictor for regression models using fine motor function, prewriting, cognitive, and expressive language as outcome variables. Therefore, regression analysis of predictors fitting to a growth curve showed that the best predictor was total intervention time, but the next best predictor was starting age, and after that, starting level of skills, depending on the outcome variable examined.(4)Rogers et al. (2012) : this study evaluated a 12-week, parent-delivered implementation of the Early Start Denver Model (ESDM) for 98 children aged 12 to 24 months. The children and families were randomized into either an ESDM treatment group or a TAU group. The intervention groups were combined to examine prediction relationships. The regression model held all other predictors constant as variables were added into the equation. Age of starting intervention and number of intervention hours significantly predicted better outcomes at Time 2 for the sample as a whole, with IQ (MSEL) as the only outcome variable tested.(5)Anderson et al. (2014)  reported on a long-term follow-up study that examined the factors at age 2 and 3 years that were associated with cognitive outcome at age 19, comparing those cognitively “more able” and “less able” (VIQ ≥ 70 or VIQ < 70 at Time 2) for 85 children diagnosed with ASD at age 2. To reduce the strong effects of Time 1 IQ when examining other types of predictive relationships, the researchers focused on two smaller groups: 32 youth with IQ ≥ 70, 24 with ASD, and eight who no longer had an ASD diagnosis (Very Positive Outcome or VPO). First, the two groups were compared on Time 1 scores to determine that they did not differ on baseline verbal IQ, nonverbal IQ, ADOS, and ADI-R scores, and thus, these features were unlikely to be confounds. Then, analyses conducted on early intervention experiences found that those who no longer had ASD were more likely to have had EI between ages 2 and 3 compared to those who were in the ASD group.(6)Vivanti et al. (2018) : this was an RCT comparing ESDM in an inclusive classroom setting (N = 22) to one that was segregated, that is, only toddlers with ASD attended (N = 22). The follow-up period was nine months (one school calendar year). The role of age (when starting the program) was investigated for each group in predicting various outcome measures, including MSEL VDQ, NVDQ, video-rated language and social measures, an imitation task, and parent-rated ASD symptoms on the SCQ and the Repetitive Behavior Questionnaire-Revised (RBQ-R) . The first set of analyses involved partial correlations between age of entry and each outcome variable, partialling out the Time 1 score for each variable. The one Time 2 variable that was significantly associated with starting age was MSEL Verbal DQ. The second approach was a linear regression. Age of starting remained a significant predictor of VDQ after the variance associated with baseline VDQ and treatment group was accounted for.
Studies demonstrating the significant beneficial predictive effect of age of starting intervention earlier versus later and generating an estimate of unique variance are as follows:(1)Perry et al. (2011) : this study included primarily preschoolers (2 to 7 years old, N = 332), participating in a community-based, early IBI program with an average length of follow-up of 18 months. The researchers computed separate regressions for eight outcome variables (cognitive, adaptive, and autism severity measures) to see whether age contributed unique variance for each. At Step 1, they entered the Time 1 score for the same variable as a way of controlling for it and reported change in R2. Then, for each of the analyses, age was entered at Step 2, noting the increase in R2 and its significance. Age of starting intervention accounted for a small, but significant, amount of unique variance for most of the outcome variables (from 1 to 6% of incremental variance). A second approach used within this study was a hierarchical regression with a simultaneous solution using the variables of age, initial IQ, VABS ABC score, and CARS total score in predicting Time 2 IQ scores. Time 1 IQ was entered first and accounted for 54% of the variance. Age at entry was then entered after baseline IQ and accounted for an additional 5.3% of the variance ().(2)Kasari et al. (2012) : this study was described above. In a separate hierarchical regression to predict spoken vocabulary at age 8, they input child age at entry to intervention (all spanned 3 to 4 years old), two early prelinguistic ability variables of initiating joint attention (IJA) and developmental play level and a treatment variable (treatment vs. no treatment). Starting age predicted by itself 14% of the variance of later spoken vocabulary (). IJA, play level, and treatment assignment also each contributed unique variance (adding up to 50%).(3)Flanagan et al. (2012) : this prediction study was carried out as a follow-up paper to previous intervention studies, including some subjects from the Perry et al. (2011)  study (see also ). Researchers reviewed the files of 61 age-matched pairs of children from a community-based EIBI program; one of each matched pair received IBI and the other was in a waitlist control group. Follow-up at Time 2 was after at least 12 months of intervention. Children in the IBI group were in treatment longer/were older at Time 2; otherwise, there were no differences between the two groups. The unique contribution of age was calculated in the context of a significant interaction. The regression analysis used one outcome variable—IQ at Time 2. The duration of treatment was entered at Step 1 because it differed between the two groups at baseline. Initial age was entered at Step 2 and was found to exert a significant effect, but with further steps, its unique contribution was diminished to nonsignificance. However, the interaction term of Initial Age × Treatment Group, at Step 7, was significant. In this context, the amount of unique variance accounted for by age was 13%.(4)Itzchak and Zachor, 2011 : this study examined 78 children aged 15–35 months and evaluated predictors of outcome apart from treatment. Two groups received either early ABA or eclectic treatment. Predictors included child variables such as chronological age (CA) when starting intervention as well as the environmental variables of mother’s age and education. Two hierarchical multiple regression analyses were performed using theoretically determined type and order of predictors. CA was included for the second regression analysis, with the change score in MSEL from Time 1 to Time 2 as the cognitive outcome. When CA was entered at the third step, it accounted for 3% () of the unique variance in the model.
Studies demonstrating the role of age of starting early intervention through significant interaction effects are as follows.
In the following studies, the researchers were able to demonstrate, through the testing of interaction effects, more complex relationships among predictors when age was included in the interaction term.(1)Flanagan et al. (2012) : this study was discussed above in the context of the percentage of unique variance of age in outcome prediction equations. The interaction term of Initial Age × Treatment Group was found to be significant, showing that younger age at initiation of treatment made a positive difference, but only if children were receiving IBI rather than TAU.(2)Vivanti et al. (2016) : the differential impact of ESDM on the outcomes of younger children (18 to 48 months old) and older children (48 to 62 months old) was examined. Children received the ESDM for one year at 20 hours per week. An analysis of covariance (ANCOVA) was performed with Time 2 outcome as the within-subjects factor and age group (cutoff: above and below 48 months when starting intervention) as the between-subjects factor. Since the age groups did differ on Nonverbal developmental quotient (DQ), this was entered as the covariate. A significant Age Group × Time interaction was found, indicating that while both groups showed a significant increase in Verbal DQ over the 12 months of treatment, children in the younger group experienced comparatively larger gains (16 DQ points versus 7 points in the older group). This effect did not apply to Nonverbal DQ.(3)Smith et al. (2015) : this study was discussed above as examining a community-based, two-year Young Autism Program with preschoolers. They entered an interaction term to evaluate the rate of progress in Year 1 and Year 2 in relation to age starting the program. A significant finding for the Age × Time interaction term showed that the progress slope was steeper for younger children, more so for Year 1 than Year 2. Therefore, age made a difference in how children benefited from the intervention: children who were younger when starting made more progress in the first year of intervention than those who were older (who also made progress, but not as much).
Studies showing that children made more gains in the beginning of the intervention (when they were younger) than in the later phases are as follows:(1)Smith et al. (2015) : this study was discussed above as examining a community-based, two-year Young Autism Program with preschoolers, using regression analyses to evaluate the role of age as an individual predictor. They also entered an interaction term to examine the rate of progress in Year 1 and Year 2, and the nature of that progress for a younger vs. older group. They found that all children made more initial progress in Year 1, with progress tapering off in Year 2, but that the younger a child was, the steeper the initial slope of progress.(2)Virues-Ortega et al. (2013) : this study used trajectory analysis and showed that the shape of the progress as shown in a learning curve was not linear for most children, but instead improved steeply in the beginning of their time in intervention and then plateaued. This applied more to children who started at a lower level of functioning than those who started with higher baseline scores. Analyzing the shape and course of the curves using “visual inspection,” it seemed evident that the higher-functioning children had steeper earlier curves and did not as frequently plateau during the intervention study time, but continued to make developmental gains. Regression analysis of predictors fitting to a growth curve showed that the best predictor was total intervention time, but the next best predictor was starting age, depending on the outcome variable. This suggests that younger-starting children were more likely to have a steeper progress curve when intervention first began compared to older-starting children.
3.3. Further Methodological Themes and Considerations
3.3.1. Different Combinations of Predictors and Outcomes Gave Varied Results
Although there was general consistency among studies in terms of the constructs/variables examined, there were considerable differences in the pairing of predictors (apart from starting age of intervention) and outcomes. Even with similar sets of outcomes, significant results for the role of starting age were quite variable. For example, Virues-Ortega et al.  showed that the age children started intervention best improved (after the amount of time in intervention) the trajectory curve fit for gross motor function, receptive language, self-care, and social behavior, but preintervention level best improved the same regression model for outcome variables of fine motor function, prewriting, cognitive, and expressive language. Vivanti et al.  found a significant Age Group × Time interaction for Verbal DQ, but not for Nonverbal DQ. The majority of the studies showed a similar variety of results. The extent to which each specific domain was an outcome area affected by age of starting EI was calculated and is presented in Table 4. Averaging across all domains, earlier age made a positive difference about half the time.
3.3.2. Specific Measurement Inconsistencies
One possible explanation for inconsistencies across studies when similar questions are asked may involve method and measurement choices. For cognitive measurement, there was a fair amount of consistency in instruments used, but then considerable inconsistency in which and how scores were employed for statistical analysis. For example, the MSEL was used in eight out of the fourteen studies. Yet five of the studies used ratio IQs made from the age equivalent scores in order to avoid the floor effect of the lowest standard score attainable being 20, while the two others used the standardized scale or composite scores from the manual tables. Across studies, there was a tendency to differentiate the MSEL verbal subscales from the nonverbal for both prediction and outcomes, yet some studies used the composite score. In community studies where there was less control over early assessment measures used, authors tended to equalize the cognitive scores by creating developmental quotients (DQs) based on age equivalents from various tests used. Most studies left out children for whom test scores could not be obtained.
Similarly, researchers made different decisions regarding specific VABS scores when addressing adaptive functioning. Some studies used age equivalents and others used standard scores; some used individual domains and others used the Adaptive Behavior Composite (ABC). There were other differences in terms of the edition used, and whether the researchers transformed the distribution or not.
Regarding autism severity, the three studies using the ADOS-T for either predictor or outcome variables each used the 1–10 severity score ratings, showing consistency in this way. In contrast, all three studies that focused on children under three years of age used the ADOS differently, ranging from the edition used to modifying the scores in an unconventional manner. Measurement of early other social and linguistic skills was highly variable, with little overlap among studies, making comparison difficult.
3.3.3. Different Approaches to Data Analysis
In addition to the use of different predictors and outcomes that were measured through various instruments using a variety of scores, data analytic strategies were highly divergent. Each study using regression analysis employed a somewhat different approach for such procedures as choosing the candidate predictors, the number of predictors and outcome variables, the regression method per se, and reporting of β and ⊗ R2.
Twelve out of the 14 studies reviewed did have at least one finding where earlier age of starting intervention was a statistically significant predictor of better developmental functioning and/or diagnostic status outcome in children with ASD. However, in the three studies where the unique variance for baseline age was reported, the amount was quite small (3% to 13%), especially in comparison with more robust predictors (usually T1 cognitive level). As well, the specific outcome measures that were predicted with statistical significance by initial age differed greatly among the studies.
More informative were findings that revealed more complex relationships among predictors when initial age was included. Three studies showed that initial age interacted with other variables, those of (1) effective treatment (younger age combined with more effective treatment (IBI) versus TAU made a positive difference in outcome); (2) more advanced initial language level (younger age combined with better early language was related to the better outcome); and (3) phase of intervention (children who were younger made more progress when they first started EI compared to older children).
This third finding—that the younger the children were, the more responsive they were to intervention, and that this response tapered off even at a young age—was evident throughout older to younger samples. Smith et al.’s (2015)  study included children as young as 20 months but primarily 3 to 5 years old, showing the effect for children who are not necessarily the youngest that participate in early intervention. The sample for Virues-Ortega et al.  was 25 months to 6 years, 5 months and produced similar effects. As the samples got younger, the effect continued to be found, for example, in four studies that included only children under 3 years of age.
Specifically, Anderson et al.’s (2014)  results indicated that receiving intervention between 24 and 36 months of age emerged as a difference between youth who no longer had an ASD diagnosis and those that retained an ASD diagnosis and had an average or above IQ. Orinstein et al.  had very similar findings, although their paper was not included in this review due to lack of control for baseline levels of several predictors. In these two studies, children generally did not start early intervention before age 24 months, but in the other two studies of the review that examined children only under age 3 years, some children were 14 to 15 months old when they started intervention. For example, children in the Itzchak and Zachor (2011)  study ranged from 15 to 35 months old, a span of 20 months, and baseline age was shown to account for a small amount of independent variance in the prediction of MSEL cognitive outcome. Rogers et al. , however, had a much younger group: 14–24 months, a spread of 10 months. Their results that age of starting intervention predicted MSEL IQ only three months later suggest that earlier is better could apply to the youngest children who are now receiving early intervention for ASD. The studies as a whole did not answer the question of when a relatively neuroplastic phase may no longer exert its influence, but they do suggest that the effect is operational in the youngest children.
Although this review intentionally focused on the variable of chronological age, attention to other predictors of outcome functioning was inevitable given that age interacted with or had relationships with these other variables. Up to this point in the literature, initial IQ or another early marker for overall ability level, possibly representing a neurological cap on developmental potential, usually has been considered to have the strongest influence on later functioning. This was the case for some of the studies in the current review [27, 31, 32]. However, in other studies that included predictors representing additional child characteristics on the one hand, and environmental variables such as type and intensity of treatment and demographics on the other hand, the overwhelming effect of initial IQ level was often mitigated. Nonetheless, the overriding effects of more severe delays could often be seen. From a methodological viewpoint, then, it will be important to take into account the number of children in a study sample with intellectual disability at Time 1, since the higher the proportion there is, the more difficult it will be to show differential prediction on other accounts.
The types of predictor variables examined in the reviewed studies were overwhelmingly child characteristics such as developmental or language level, adaptive behavior, social-communication skill levels, and autism severity, as well as child initial age. Yet child-focused/biologic variables and environmental risk and protective features are assumed to operate in a dynamic interplay, supporting or undermining development depending on timing, strength, and chronicity. The one environmental variable that was measured by all studies was treatment, since the studies were selected to include this feature. Several of the studies assessed parental variables such as educational and income level for the purposes of reporting sample characteristics and to demonstrate that groups being compared did not differ on these features. Only three of the studies included another environmental variable besides treatment as a predictor. Both Rogers et al.  and Eapen et al.  included a proximal parental variable reflecting treatment fidelity (e.g., how effectively the mother learned and used the interaction style that was taught during intervention) and it was found to influence the outcome positively. Itzchak and Zachor (2011)  found that the more distal variables of maternal age and education were significant predictors.
4.1. Other Key Methodological Issues
In 11 out of 14 of the studies, several different combinations of predictor and outcome variables were explored, with very inconsistent results across studies. What accounts for different positive versus negative findings given which predictors and which outcome variables were examined undoubtedly relates to both validity and reliability features of the measurement choices. A closer look at the specific scores used to represent the key constructs suggested that certain technical choices may be important for detecting reliable changes and true statistical relationships. For example, several authors have made detailed arguments for the use of age equivalents, especially in VABS adaptive behavior scores [32, 66].
For both VABS and MSEL, authors made different choices for examining subdomains rather than using composite scores.
Studies with young children with ASD increasingly differentiate verbal vs. nonverbal scores, with good reason, since language ability can be dissociated from the overall cognitive level across typical and disability populations. Studies also have shown the value of treating VABS domains (communication, daily living skills, etc.) as having different trajectories in autism populations presumably because the domains tap into different ability areas [66–68].
4.2. Implications for Treatment
The positive findings in the reviewed studies for “the earlier, the better” have implications for continued efforts for early identification and treatment. Even in the face of negative findings, there is a clear rationale for involving families with children with disabilities as early as possible in terms of support and advocacy skills that involvement with the intervention systems affords . Most children with autism spectrum disorder will require first educational services and then vocational and adult support services. Therefore, the earlier caregivers learn what their children’s needs are, what services they benefit from, and how the service systems work, the more able they will be to make decisions to support optimal development for their individual child .
Furthermore, the findings in these studies based on interactions among child characteristics have direct relevance to the current focus in the intervention literature regarding heterogeneous response to intervention. Translational questions have been posed regarding the difference between “rapid learners” vs. those with “poor response” , those who gained the most and lost the most , and responders vs. nonresponders . Related to this is the question of which intervention approach is appropriate for which children. The themes emerging in the review regarding preintervention skill level, evidence-based intervention strategies, and amount of treatment, in addition to age at initiation of intervention, point to considerations for these important questions.
5. Conclusions and Future Directions
It is our assertion that this collection of studies and the current review constitutes a beginning evidence base for “earlier is better” with regard to interventions for young children with ASD. Table 5 shows recommendations for future efforts that are examining whether earlier treatment initiation gives children with ASD an advantage in terms of later outcome. It is clear that this focus of inquiry will benefit from further research that is informed by these findings, as well as by the methodological strengths and limitations of the reviewed studies.
Conflicts of Interest
The authors declare they have no conflicts of interest.
This research was supported in part by Grant no. T73MC29073 from the Health Resources and Services Administration.
- J. Baio, L. Wiggins, D. L. Christensen et al., “Prevalence of autism spectrum disorder among children aged 8 Years - autism and developmental disabilities monitoring network, 11 sites, United States, 2014,” MMWR. Surveillance Summaries, vol. 67, no. 6, pp. 1–23, 2018.
- K. Kurzius-Spencer, R. Paul, A. Klin, S. Hannigen, L. E. Dichtel, and F. Volkmar, “Parental recognition of developmental problems in toddlers with autism spectrum disorders,” Journal of Autism and Developmental Disorders, vol. 37, no. 1, pp. 62–72, 2007.
- L. C. Eaves and H. H. Ho, “The very early identification of autism: outcome to age 4½-5,” Journal of Autism and Developmental Disorders, vol. 34, no. 4, pp. 367–378, 2004.
- L. Zwaigenbaum, M. L. Bauman, R. Choueiri et al., “Early interventions for children with autism spectrum disorder under 3 years of age: recommendations for practice and research,” Pediatrics, vol. 136, pp. 60–81, 2015.
- Z. Warren, M. L. McPheeters, N. Sathe, J. H. Foss-Feig, A. Glasser, and J. Veenstra-VanderWeele, “A systematic review of early intensive intervention for autism spectrum disorders,” Pediatrics, vol. 127, no. 5, pp. e1303–e1311, 2011.
- B. Reichow, E. E. Barton, B. A. Boyd, and K. Hume, “Early intensive behavioral intervention (EIBI) for young children with autism spectrum disorders (ASD): a systematic review,” Campbell Systematic Reviews, vol. 9, 2014.
- A. Estes, J. Munson, S. J. Rogers, J. Greenson, J. Winter, and G. Dawson, “Long-term outcomes of early intervention in 6-year-old children with autism spectrum disorder,” Journal of the American Academy of Child & Adolescent Psychiatry, vol. 54, no. 7, pp. 580–587, 2015.
- Z. Cidav, L. Lawer, S. C. Marcus, and D. S. Mandell, “Age-related variation in health service use and associated expenditures among children with autism,” Journal of Autism and Developmental Disorders, vol. 43, no. 4, pp. 924–931, 2013.
- T. A. Lavelle, M. C. Weinstein, J. P. Newhouse, K. Munir, K. A. Kuhlthau, and L. Prosser, “Economic burden of childhood autism spectrum disorders,” Pediatrics, vol. 133, 2014.
- S. J. Rogers and L. Vismara, “Interventions for infants and toddlers at risk for autism spectrum disorder,” in Handbook of Autism and Pervasive Developmental Disorders, F. R. Volkmar, S. J. Rogers, R. Paul, and K. Pelphrey, Eds., vol. 2, John Wiley & Sons, New York, NY, USA, 4th edition, 2014.
- G. Vivanti, M. Prior, K. Williams, and C. Dissanayake, “Predictors of outcomes in autism early intervention: why don’t we know more?” Frontiers in Pediatrics, vol. 2, pp. 1–10, 2014.
- B. Kolb and R. Gibb, “Brain plasticity and behaviour in the developing brain,” Journal of the Canadian Academy of Child & Adolescent Psychiatry, vol. 20, pp. 265–276, 2011.
- M. H. Johnson, “Interactive specialization: a domain-general framework for human functional brain development?” Developmental Cognitive Neuroscience, vol. 1, no. 1, pp. 7–21, 2011.
- D. Rice and S. Barone, “Critical periods of vulnerability for the developing nervous system: evidence from humans and animal models,” Environmental Health Perspectives, vol. 108, pp. 511–533, 2000.
- J. Barbaro and C. Dissanayake, “Autism spectrum disorders in infancy and toddlerhood: a review of the evidence on early signs, early identification tools, and early diagnosis,” Journal of Developmental & Behavioral Pediatrics, vol. 30, no. 5, pp. 447–459, 2009.
- R. J. Landa, K. C. Holman, and E. Garrett-Mayer, “Social and communication development in toddlers with early and later diagnosis of autism spectrum disorders,” Archives of General Psychiatry, vol. 64, no. 7, pp. 853–864, 2007.
- N. Yirmiya and T. Charman, “The prodrome of autism: early behavioral and biological signs, regression, peri- and post-natal development and genetics,” Journal of Child Psychology and Psychiatry, vol. 51, no. 4, pp. 432–458, 2010.
- J. Piven, J. T. Elison, and M. J. Zylka, “Toward a conceptual framework for early brain and behavior development in autism,” Molecular Psychiatry, vol. 22, no. 10, pp. 1385–1394, 2017.
- S. Bottini, “Social reward processing in individuals with autism spectrum disorder: a systematic review of the social motivation hypothesis,” Research in Autism Spectrum Disorders, vol. 45, pp. 9–26, 2018.
- C. Chevallier, G. Kohls, V. Troiani, E. S. Brodkin, and R. T. Schultz, “The social motivation theory of autism,” Trends in Cognitive Sciences, vol. 16, no. 4, pp. 231–239, 2012.
- G. Dawson, “Early behavioral intervention, brain plasticity, and the prevention of autism spectrum disorder,” Development and Psychopathology, vol. 20, no. 3, pp. 775–803, 2008.
- C. C. Clements, A. R. Zoltowski, L. D. Yankowitz, B. E. Yerys, R. T. Schultz, and J. D. Herrington, “Evaluation of the social motivation hypothesis of autism,” JAMA Psychiatry, vol. 75, no. 8, pp. 797–808, 2018.
- T. Gliga, E. J. H. Jones, R. Bedford, T. Charman, and M. H. Johnson, “From early markers to neuro-developmental mechanisms of autism,” Developmental Review, vol. 34, no. 3, pp. 189–207, 2014.
- A. S. Masten and D. Cicchetti, “Developmental cascades,” Development and Psychopathology, vol. 22, pp. 491–495, 2010.
- A. Klin, S. Shultz, and W. Jones, “Social visual engagement in infants and toddlers with autism: early developmental transitions and a model of pathogenesis,” Neuroscience & Biobehavioral Reviews, vol. 50, pp. 189–203, 2015.
- J. L. Matson and K. R. M. Smith, “Current status of intensive behavioral interventions for young children with autism and PDD-NOS,” Research in Autism Spectrum Disorders, vol. 2, no. 1, pp. 60–74, 2008.
- A. Perry, A. Cummings, J. D. Geier et al., “Predictors of outcome for children receiving intensive behavioral intervention in a large, community-based program,” Research in Autism Spectrum Disorders, vol. 5, no. 1, pp. 592–603, 2011.
- E. Ben-Itzchak, L. R. Watson, and D. A. Zachor, “Cognitive ability is associated with different outcome trajectories in autism spectrum disorders,” Journal of Autism and Developmental Disorders, vol. 44, no. 9, pp. 2221–2229, 2014.
- D. A. Zachor and E. Ben Itzchak, “Variables affecting outcome of early intervention in autism spectrum disorder,” Journal of Pediatric Neurology, vol. 15, pp. 129–133, 2017.
- T. Charman, E. Taylor, A. Drew, H. Cockerill, J.-A. Brown, and G. Baird, “Outcome at 7 years of children diagnosed with autism at age 2: predictive validity of assessments conducted at 2 and 3 years of age and pattern of symptom change over time,” Journal of Child Psychology and Psychiatry, vol. 46, no. 5, pp. 500–513, 2005.
- C. Kasari, A. Gulsrud, S. Freeman, T. Paparella, and G. Hellemann, “Longitudinal follow-up of children with autism receiving targeted interventions on joint attention and play,” Journal of the American Academy of Child & Adolescent Psychiatry, vol. 51, no. 5, pp. 487–495, 2012.
- T. Smith, R. Klorman, and D. W. Mruzek, “Predicting outcome of community-based early intensive behavioral intervention for children with autism,” Journal of Abnormal Child Psychology, vol. 43, no. 7, pp. 1271–1282, 2015.
- B. Zablotsky, L. J. Colpe, B. A. Pringle, M. D. Kogan, C. Rice, and S. J. Blumberg, “Age of parental concern, diagnosis, and service initiation among children with autism spectrum disorder,” American Journal on Intellectual and Developmental Disabilities, vol. 122, no. 1, pp. 49–61, 2017.
- A. Goodwin, N. L. Matthews, and C. J. Smith, “Parent-reported early symptoms of autism spectrum disorder in children without intellectual disability who were diagnosed at school age,” Autism, vol. 23, no. 3, pp. 770–782, 2018.
- C. Fountain, M. D. King, and P. S. Bearman, “Age of diagnosis for autism: individual and community factors across 10 birth cohorts,” Journal of Epidemiology & Community Health, vol. 65, no. 6, pp. 503–510, 2011.
- A. M. Daniels and D. S. Mandell, “Explaining differences in age at autism spectrum disorder diagnosis: a critical review,” Autism, vol. 18, no. 5, pp. 583–597, 2014.
- E. M. Mullen, Infant Mullen Scales of Early Learning, American Guidance Service, Circle Pines, MN, USA, 1995.
- C. D. Elliott, Differential Ability Scales, Psychological Corporation, San Antonio, TX, USA, 1990.
- C. D. Elliott, Differential Ability Scales, Harcourt Assessment, San Antonio, TX, USA, 2nd edition, 2007.
- D. Wechsler, Wechsler Intelligence Scale for Children, Psychological Corporation, San Antonio, TX, USA, 4th edition, 2012.
- S. S. Sparrow, D. Balla, and D. V. Cicchetti, Vineland Adaptive Behavior Scales, American Guidance Service, Circle Pines, MN, USA, 1984.
- S. S. Sparrow, D. V. Cicchetti, and D. A. Balla, Vineland Adaptive Behavior Scales: Second Edition (Vineland II), Survey Interview Form/caregiver Rating Form, Pearson Assessments, Livonia, MN, USA, 2005.
- C. Lord, M. Rutter, P. DiLavore, S. Risi, K. Gotham, and S. Bishop, Autism Diagnostic Observation Schedule, Western Psychological Corporation, Los Angeles, CA, USA, 2nd edition, 2012.
- E. Schopler, R. J. Reichler, and B. R. Renner, The Childhood Autism Rating Scale (CARS), Western Psychological Services, Los Angeles, CA, USA, 1988.
- M. L. Rutter, A. Bailey, and C. Lord, Social Communication Questionnaire (SCQ), Western Psychological Services, Los Angeles, CA, USA, 2003.
- Å. Hedvall, J. Westerlund, E. Fernell et al., “Preschoolers with autism spectrum disorder followed for 2 years: those who gained and those who lost the most in terms of adaptive functioning outcome,” Journal of Autism and Developmental Disorders, vol. 45, no. 11, pp. 3624–3633, 2015.
- M. Tiura, J. Kim, D. Detmers, and H. Baldi, “Predictors of longitudinal ABA treatment outcomes for children with autism: a growth curve analysis,” Research in Developmental Disabilities, vol. 70, pp. 185–197, 2017.
- G. Alpern, Developmental Profile-3, Western Psychological Corporation, Los Angeles, CA, USA, 2009.
- V. Eapen, R. Crncec, and A. Walter, “There are gains, but can we tell for whom and why? Predictors of treatment response following group Early Start Denver Model intervention in preschool-aged children with autism spectrum disorder,” Autism Open Access, vol. 6, p. 168, 2016.
- C. Kasari, S. Freeman, and T. Paparella, “Joint attention and symbolic play in young children with autism: a randomized controlled intervention study,” Journal of Child Psychology and Psychiatry, vol. 47, no. 6, pp. 611–620, 2006.
- C. Kasari, T. Paparella, S. Freeman, and L. B. Jahromi, “Language outcome in autism: randomized comparison of joint attention and play interventions,” Journal of Consulting and Clinical Psychology, vol. 76, no. 1, pp. 125–137, 2008.
- K. T. Williams, Expressive Vocabulary Test (EVT), American Guidance Services, Inc., Circle Pines, MN, USA, 1997.
- J. Virues-Ortega, V. Rodríguez, and C. T. Yu, “Prediction of treatment outcomes and longitudinal analysis in children with autism undergoing intensive behavioral intervention,” International Journal of Clinical and Health Psychology, vol. 13, no. 2, pp. 91–100, 2013.
- M. E. Glover, A. R. Preminger, and A. R. Sanford, Early LAP: The Early Learning Accomplishment Profile for Young Children: Birth to 36 Months, Chapel Hill Training-Outreach Project, Chapel Hill, NC, USA, 1988.
- E. S. Peisner-Feinberg and B. J. Hardin, The Early Learning Accomplishment Profile Edition Examiner’s Manual and Technical Report, Kaplan, New York, NY, USA, 2001.
- B. J. Hardin, E. S. Peisner-Feinberg, and S. W. Weeks, The Learning Accomplishment Profile-Diagnostic (LAP-D), Kaplan Early Learning, Lewisville, NC, USA, 3rd edition, 2005.
- S. J. Rogers, A. Estes, C. Lord et al., “Effects of a brief early start denver model (ESDM)-Based parent intervention on toddlers at risk for autism spectrum disorders: a randomized controlled trial,” Journal of the American Academy of Child & Adolescent Psychiatry, vol. 51, no. 10, pp. 1052–1065, 2012.
- D. K. Anderson, J. W. Liang, and C. Lord, “Predicting young adult outcome among more and less cognitively able individuals with autism spectrum disorders,” Journal of Child Psychology and Psychiatry, vol. 55, no. 5, pp. 485–494, 2014.
- G. Vivanti, C. Dissanayake, E. Duncan et al., “Outcomes of children receiving Group-Early Start Denver Model in an inclusive versus autism-specific setting: a pilot randomized controlled trial,” Autism, vol. 23, no. 5, pp. 1165–1175, 2018.
- J. W. Bent, F. J. Symons, D. E. Parker, and M. H. Lewis, “Varieties of repetitive behavior in autism: comparisons to mental retardation,” Journal of Autism and Developmental Disorders, vol. 30, no. 3, pp. 237–243, 2000.
- H. E. Flanagan, A. Perry, and N. L. Freeman, “Effectiveness of large-scale community-based Intensive Behavioral Intervention: a waitlist comparison study exploring outcomes and predictors,” Research in Autism Spectrum Disorders, vol. 6, no. 2, pp. 673–682, 2012.
- N. L. Freeman and A. Perry, “Outcomes of intensive behavioural intervention in the toronto preschool autism service,” Journal on Developmental Disabilities, vol. 16, pp. 17–32, 2010.
- E. B. Itzchak and D. A. Zachor, “Who benefits from early intervention in autism spectrum disorders?” Research in Autism Spectrum Disorders, vol. 5, pp. 345–358, 2011.
- G. Vivanti, C. Dissanayake, and The Victorian ASELCC Team, “Outcome for children receiving the early start denver model before and after 48 months,” Journal of Autism and Developmental Disorders, vol. 46, no. 7, pp. 2441–2449, 2016.
- A. J. Orinstein, M. Helt, E. Troyb et al., “Intervention for optimal outcome in children and adolescents with a history of autism,” Journal of Developmental & Behavioral Pediatrics, vol. 35, no. 4, pp. 247–256, 2014.
- S. Naigles, J. M. Paynter, and L. Gilmore, “Vineland adaptive behavior scales: II profile of young children with autism spectrum disorder,” Journal of Autism and Developmental Disorders, vol. 46, no. 1, pp. 64–73, 2016.
- D. Kraijer, “Review of adaptive behavior studies in mentally retarded persons with autism/pervasive developmental disorder,” Journal of Autism and Developmental Disorders, vol. 30, no. 1, pp. 39–47, 2000.
- A. Perry, H. E. Flanagan, J. Dunn Geier, and N. L. Freeman, “Brief report: the Vineland Adaptive Behavior Scales in young children with autism spectrum disorders at different cognitive levels,” Journal of Autism and Developmental Disorders, vol. 39, no. 7, pp. 1066–1078, 2009.
- H. H. Schertz, C. Baker, C. Baker, S. Hurwitz, and L. Benner, “Principles of early intervention reflected in toddler research in autism spectrum disorders,” Topics in Early Childhood Special Education, vol. 31, no. 1, pp. 4–21, 2011.
- R. S. Tolmie, S. Bruck, and R. Kerslake, “The early intervention readiness program (EIRP),” Topics in Early Childhood Special Education, vol. 36, no. 4, pp. 242–250, 2017.
- R. J. Landa, “Efficacy of early interventions for infants and young children with, and at risk for, autism spectrum disorders,” International Review of Psychiatry, vol. 30, no. 1, pp. 25–39, 2018.
- K. Fossum, L. Williams, N. Garon, S. E. Bryson, and I. M. Smith, “Pivotal response treatment for preschoolers with autism spectrum disorder: defining a predictor profile,” Autism Research, vol. 11, 2018.
Copyright © 2020 Patricia O. Towle et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.