Abstract
To assess the effectiveness of the Tier 1 Program of the Project P.A.T.H.S., a randomized group trial with eight waves of data collected was carried out. At the fifth year of data collection, 19 experimental schools (, 662 students) and 24 control schools (, 272 students) participated in the study. Analyses based on individual growth curve modeling showed that participants in the experimental schools displayed better positive youth development than did participants in the control schools in terms of different indicators derived from the Chinese Positive Youth Development Scale, including moral competence and behavioral competence and cognitive behavioral competencies. Significant results were also found when examining the trajectories of psychological development among control and experimental participants who perceived the program to be beneficial. Findings based on longitudinal objective outcome evaluation strongly suggest that the Project P.A.T.H.S. is effective in promoting positive development in Hong Kong secondary school students.
1. Introduction
Adolescence is an age of transition. With physical and cognitive maturation taking place in puberty, intrapersonal, and interpersonal changes in adolescents intensify and there are growing social demands and expectations during adolescence. Because of these changes, adolescence is also regarded as an age of stress. The stressors confronting an adolescent might include family stressors (e.g., parental conflict, parental marital problems), interpersonal stressors (e.g., no friends), academic stress (e.g., examination stress), living circumstances-related stressors (e.g., immigration), financial stressors (e.g., poverty), developmental stressors (e.g., early or late maturing), psychological stressors (e.g., lack of life meaning), and social stressors (e.g., high competition). In a Chinese society such as Hong Kong, stress arising from academic excellence and subtle social competition is particularly relevant for Chinese adolescents. Obviously, how to cope with stress in adolescence is an important developmental task for adolescents.
At the same time, there are several myths about development of adolescents in the Chinese culture [1]. These include as follows (a) young people will grow up automatically, (b) young people are usually troublesome, (c) students in schools with good academic achievement do not have problems, (d) problem free is healthy development, (e) we should focus our attention on the solving of adolescent problems, (f) different adolescent developmental problems require different solutions, and (g) adolescent developmental problems are the sole problem of the government. With reference to these myths, it is argued that (a) adolescent development requires nurturance, (b) young people have strengths and potentials, (c) students attending schools with good academic achievement also display problems, (d) problem free is not fully prepared, (e) solving adolescent problems and prevention are equally important, (f) different adolescent developmental problems have similar root causes and prevention methods, and (g) adolescent development is a topic that is owned by different stakeholders in the society.
One obvious way to nurture young people is to promote social and emotional competencies of young people [2]. According to the Collaborative for Academic, Social and Emotional Learning (CASEL), โsocial and emotional learning (SEL) is the process of acquiring the skills to recognize and manage emotions, develop caring and concern for others, make responsible decisions, establish positive relationships, and handle challenging situations effectively. Research has shown that SEL is fundamental to childrenโs social and emotional developmentโtheir health, ethical development, citizenship, academic learning, and motivation to achieve. Social and emotional education is a unifying concept for organizing and coordinating school-based programming that focuses on positive youth development, health promotion, prevention of problem behaviors, and student engagement in learningโ (http://www.casel.org/). Generally speaking, several SEL attributes are commonly included in different SEL models. These include self-awareness (identifying emotions and recognizing strengths), social awareness (perspective-taking and appreciating diversity), self-management (managing emotions and goal setting), responsible decision making (analyzing situations, assuming personal responsibility, respecting others, and problem solving), and relationship skills (communication, building relationships, negotiation, refusal). Sun and Shek [3] showed that higher level of positive youth development predicted lower level of problem behavior, thus suggesting that positive youth development is an important protective factor in adolescent problem behavior. Emphasis on the importance of SEL is strong in North America and some Asian countries such as Singapore.
How is SEL implemented in Hong Kong? Instead of focusing on life skills and competencies in students, moral education focusing on values has been emphasized in Hong Kong. While the policy and curriculum guide governing moral education are elegant, there are three problems in the related policy and its actual implementation. First, there are no curriculum materials which have been validated, although there is a pool of suggested curriculum materials that can be used by teachers. Second, there is a wide variation in the mode of implementation of moral education in schools settings. While some schools may incorporate moral education in the formal curriculum, some may use extracurricular activities such as morning assemblies to implement moral education. Finally, rigorous evaluation of moral education programs is rare, although there are administrative audits and collection of management information by the government. The lack of rigorous evaluation implies that there is no way to understand whether there are changes in outcomes as such moral values of the students because input and output evaluation is simply not adequate.
Shek and Yu [4] reviewed adolescent prevention and positive youth development programs in Asia which have been evaluated by studies adopting true experimental or quasiexperimental designs. They found that compared to Western societies, the number of validated programs in different Asian communities was terribly low. Also, there were comparatively more programs addressing substance abuse than other mental health problems. Compared to evaluated prevention programs, there were very few positive youth development programs. Finally, there were very few rigorously designed evaluative studies of prevention and positive youth development programs over a long period of time. The lack of adolescent prevention and positive youth development programs in Asia has three implications. First, the findings suggest that we lack evidence-based solutions to adolescent developmental issues. Second, the findings mean that we do not clearly know the benefits and harms of the existing programs. Third, the lack of studies also means that there is no accountability of the workers. Against this background, it is important to conduct more evaluation studies for positive youth development programs in Hong Kong.
The Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programs) is a youth enhancement program that attempts to promote holistic youth development in Hong Kong. While the Tier 1 Program is a universal positive youth development program adopting a curricular-based approach for students in Secondary 1 to Secondary 3, the Tier 2 program is designed for students with greater psychosocial needs. As far as objective outcome evaluation is concerned, several studies have showed that students who participated in the project showed better development than those who did not participate. Based on the first two waves collected in a randomized group trial, Shek et al. [5] showed that participants in the experimental schools had significantly higher positive youth development levels than those in the control schools. By using the first four waves of data collected in the first two years of the Full Implementation Phase, analyses based on generalized linear models and linear mixed methods similarly showed that students in the experimental schools generally developed better than those in the control schools [6, 7].
With reference to the first six waves of data in the junior secondary school years (i.e., Secondary 1 to Secondary 3), evaluation findings showed that the Project P.A.T.H.S. was able to promote holistic development in the participants. Individual growth curve analyses showed that participants in the experimental schools displayed better positive youth development than did participants in the control schools based on different indicators derived from the Chinese Positive Youth Development Scale, including positive self-identity, prosocial behavior, and general positive youth development attributes. Differences between experimental and control participants were also found when students joining the Tier 1 Program and perceiving the program to be beneficial were employed as participants of the experimental schools [8]. Similarly, longitudinal analyses showed that the participants in the experimental schools displayed lower levels of substance abuse and delinquent behavior than did the control school students. Participants who regarded the program to be helpful also showed lower levels of problem behavior than did the control school students [9]. Similar patterns of findings were observed with the inclusion of the seventh wave of data for analyses [10].
To replicate the objective outcome evaluation findings and to examine the long-term effectiveness of the Project P.A.T.H.S. over a period of five years, the Wave 8 data were included in the present study. Essentially, we asked whether the program effect could be sustained over a period of two years after the termination of the project at Secondary 3. In the realm of science, replication plays an important role. As pointed out by Campbell and Stanley [11], โwe must increase our time perspective, and recognize that continuous, multiple experimentation is more typical of science than once-and-for-all definitive experiments. The experiments we do today, if successful, will need replication and cross-validation at other times under other conditions before they can become an established part of science, before they can be theoretically interpreted with confidence (p. 3).โ As longitudinal evaluation studies using objective outcome indicators are rare in different Chinese context, the present study is a pioneer and ground-breaking addition to the literature.
2. Methods
2.1. Participants and Procedures
During 2006โ2011, a total of 7,846 Secondary 1 students (equivalent to Grade 1) were recruited from 48 schools. Shek and associates [12, 13] described the procedures and criteria for recruiting the initial 24 experimental schools and 24 control schools.
Students were measured at baseline in the fall of 2006 (Wave 1) and then followed longitudinally across waves (Wave 2: Spring 2007; Wave 3: Fall 2007; Wave 4: Spring 2008; Wave 5: Fall 2008; Wave 6: Spring 2009; Wave 7: Spring 2010; Wave 8: Spring 2011). In Year 1 (2006-2007), one school withdrew after Wave 1. In Year 2 (2007-2008), Waves 3 and 4 data were collected from the same cohort, with 20 experimental schools (i.e., three schools withdrew after Wave 2) and 24 control schools. In Year 3 (2008-2009), Waves 5 and 6 data were collected from the same cohort with 19 experimental schools (i.e., one experimental school dropped out after Wave 4) and 24 control schools. A total of 3,820 students completed all 8 waves of the study (49%). In the present study, all data were tested as individual growth curve model allows unequal interval spaced time points and missing data [14]. The number of completed questionnaires collected in each measurement occasion can be seen in Table 1.
At pre- and posttest, the purpose of the study was mentioned, and confidentiality of the collected data was repeatedly emphasized to all students in attendance on the day of testing. Parental and student consent had been obtained prior to data collection. All participants responded to all scales in the questionnaire in a self-administration format. Adequate time was provided for the participants to complete the questionnaire. A trained research assistant was present throughout the administration process.
2.2. Instruments
Consistent with the procedures used in Year 1, the participants were invited to respond to a questionnaire that comprised different measures of youth development at pretest (i.e., before the program began) and posttest (i.e., after the program ended). The following measures were used.
2.2.1. Chinese Positive Youth Development Scale (CPYDS)
Based on the analyses conducted in Year 1, the item composition of the 15 subscales of the CPYDS is as follows.(1)Bonding Subscale (six items).(2)Resilience Subscale (six items).(3)Social Competence Subscale (seven items).(4)Emotional Competence Subscale (six items).(5)Cognitive Competence Subscale (six items).(6)Behavioral Competence Subscale (modified five items).(7)Moral Competence Subscale (six items).(8)Self-Determination Subscale (five items).(9)Self-Efficacy Subscale (modified two items).(10)Beliefs in the Future Subscale (modified three items).(11)Clear and Positive Identity Subscale (seven items).(12)Spirituality Subscale (seven items).(13)Prosocial Involvement Subscale (five items).(14)Prosocial Norms Subscale (five items).(15)Recognition for Positive Behavior Subscale (four items).
As mentioned by Shek [1], different composite indices derived from the scale were used to assess positive youth development. First, the mean of the total mean score based on 12 subscales (excluding behavioral competence, self-determination, and prosocial norms) could be used as an overall measure of positive youth development (CPYDS-12). Next, as it can be argued that constructs including spirituality, prosocial norms, prosocial involvement, bonding, and recognition for positive behavior are different from the rest of the scales, a summation of 10 subscales (CPYDS-10) assessing psychosocial competence and strengths was used (i.e., resilience, social competence, emotional competence, cognitive competence, behavioral competence, moral competence, self-determination, self-efficacy, beliefs about the future, and clear and positive identity). Third, based on conceptual analyses of the items, one key item was derived for each domain which resulted in a 15-item key measure (KEY 15). Fourth, based on item analysis, a 36-item measure was derived for each domain (KEY 36). Fifth, based on item analysis, a 7-item measure was derived for behavioral competence and moral competence (CPYDS-2). Lastly, Shek and Ma [15] also showed that the 15 subscales in the CPYDS could be further reduced to four dimensions, including cognitive-behavioral competencies (CBC), prosocial attributes (PA), positive identity (PID) and general positive youth development qualities (GPYDQ). In general, higher scores of these variables suggested better positive youth development. The internal consistency of these measures can be seen in Table 2.
2.2.2. Data Analytic Strategies
Individual growth curve (IGC) is an advanced statistical technique which is conducted to examine โaggregatesโ of individual curves rather than separate analysis of each individual growth curve [14]. This method models individual change over time, determines the shape of the growth curves, explores systematic differences in change, and examines the effects of covariates (e.g., treatment) on group differences in the initial status and the rate of growth. Previous literature shows that this method is commonly used in the field [16, 17].
IGC is an appropriate approach in studying individual change as it creates a two-level hierarchical model that nested time within individual [18โ20]. The Level 1 model refers to the within-person or intraindividual change model (i.e., repeated measurements over time). It focuses on the individual and describes the developmental changes for each individual (i.e., the variation within individual over time). Level 1 model estimates the average within-person initial status and rate of change over time. No predictors are included in this model. The basic linear growth model is as showed below.
Level 1 model:
In our study, is the initial status (i.e., Wave 1) of the outcome variable for individual . is the linear rate of change for individual and is the residual in the outcome variable for individual at Time . is the repeatedly measured of the outcome variable for an individual at Time .
To test a nonlinear individual growth trajectory across time, other higher-order polynomial trends (i.e., quadratic and cubic slopes) can also be included for model testing. This is showed in (2), in which Time (i.e., the linear slope,) remains, while Time2 (i.e., quadratic slope, ) and Time3 (i.e., cubic slope, ) are added in the model:
The Level 2 model captures whether the rate of change vary across individuals in a systematic way. The growth parameters (i.e., the within-subjects intercepts and slope) of Level 1 are the outcome variables to be predicted by the between-subjects variables at Level 2. At this level (In (3)), an explanatory variable (such as, group in the present study) is included to analyze the predictorโs effect on interindividual variation of outcome variable. The errors are assumed to be independent and normally distributed and that the variance is equal across individuals [19].
The Level 2 model is
In our study, is the grand mean for the outcome variable for the whole sample at Time . is the initial status of the outcome variable for the whole sample at Time . is the linear slope of change relating to the outcome variable for the whole sample at Time . is the quadratic slope of change relating to the outcome variable for the whole sample at Time . is the cubic slope of change relating to the outcome variable for the whole sample at Time . , , , , are used to test whether the predictor (i.e., group) is associated with the initial status, linear growth, quadratic growth, and cubic growth, respectively. , ,โand are the residual errors that is not explained by Level 2 predictors.
In this study, we tested whether treatment was predictive of studentsโ growth parameters (i.e., initial status, linear change, quadratic change, and cubic change) in several positive youth development indicators across time. In particular, the relationships between these indicators and group were estimated after controlling the effect of gender and initial age. The intercept (i.e., initial status) and linear slope were allowed to vary across individuals. To examine the amount of total variation in the outcome variables that is related to between-individual differences, the intraclass correlation coefficient (ICC) is calculated.
A dummy variable was created (i.e., groupโcontrol versus experimental groups) as a predictor. Participants in the control group were coded as โ1 and those in the experimental group as 1. Two covariates (i.e., gender and initial age) were included when examining the predictive program effect on the outcome variables. Gender was coded as and . Similar coding method for a dichotomous variable was found in previous studies [18, 20]. For the continuous variables, grand mean centering method was generally recommended in order to simplify the interpretation of the results [21]. In our study, the mean age was 12. Initial age was then centered by subtracting the mean age, and therefore, the centered initial age was generated.
Following the strategy suggested by Singer and Willet [14], a series of models were tested. These included the following: (a) an unconditional model was tested to calculate the ICC, (b) an unconditional growth model served as a baseline model to explore whether the growth curves are linear or curvilinear, (c) two higher order polynomial models were estimated to determine if the rate of change accelerated or decelerated across time, and (d) a conditional model was formed to investigate whether the predictor was related to the growth parameters (i.e., initial status, linear growth, quadratic growth, and cubic growth). The intercept and linear slope were allowed to vary across individuals. Missing data were handled through likewise deletion.
To facilitate the interpretations of the significant interaction effects, we plotted prototypical trajectories as suggested by Singer and Willett [14] in order to demonstrate the effect of treatment on the rate of change across time. The step in creating prototypical plots is generally identical to the method of plotting graphs in regression [22]. For each outcome variable, a linear mixed model (LMM) via SPSS with maximum likelihood estimation was conducted. As we focused on the entire model (both fixed and random effects), maximum likelihood (ML) method was used [21]. The procedures for analyzing longitudinal data via SPSS can be seen in Shek and Ma [23].
3. Results
Tables 3 and 4 present the IGC findings based on several indicators derived from the CPYDS. As can be seen from the tables, there were significant treatment effects across time. Group was a significant predictor of all growth parameters (i.e., the initial status, linear, quadratic, and cubic slopes) in three outcome variables (i.e., moral competence, CPI-1, and CPYDS-2). Both groups had different initial status at the beginning (MC: , , ; CPI-1: , , ; CPYDS-2: , , ). Control group dropped slower (linear slope: , , ; CPI-1: , , ; CPYDS-2: , , ; cubic slope: , , ; CPI-1: , , ; CPYDS-2: , , ), but decelerated faster than did the experimental group (MC: , , ; CPI-1: , , ; CPYDS-2: , , ) across 8 waves. Similar trend that was also found in BC, except the test of group differences in linear slope was not significant (, , ). These results revealed that both groups differed in their rates of growth over time and these differences occurred up through Wave 7 after which they diminished gradually (see Figures 1, 2, 3, and 4).
Additional analyses were performed to examine the positive treatment effects by comparing the control group and experimental participants who found the program to be beneficial. More significant findings were shown in these analyses. Control group decreased faster and decelerated slower than did the experimental group (Tables 3 and 4). These patterns of change were shown in Figures 5, 6, 7, 8, 9, 10, 11, and 12. In general, these findings suggested that stable trajectories of positive youth development indicators were found in the experimental group, but not in the control group. These findings supported the beneficial treatment effect on participantsโ psychological development over time.
The values of ICC ranged fromโโ.36 toโโ.64 (Tables 5 and 6), indicating the nested structure of the data [24, 25]. This also suggested that over 36% of the total variation in all variables was related to individual differences. To explore the effects of treatment on all outcome variables, the amount of variance in relation to the initial status and linear slope was examined. Based on the reduction of total variance from Model 1 (M1: baseline growth model) and Model 2 (M2: model with predictors only), treatment had stronger predictive effects in the within-individual variance, but lower in the between-individual variances. It is noteworthy that these results did not change much after entering the initial age and gender as covariates (Model 3: model with predictor and covariates) (Tables 5 and 6). Lastly, based on Feingoldโs [26] suggestions, the effect sizes of all IGC models were calculated. The effect sizes ranged from low to moderate (linear slope:โโ.00 toโโ.36; quadratic slope:โโ.00 toโโ.17; cubic slope:โโ.00 toโโ.02).
4. Discussion
Amongst different evaluation strategies, objective outcome evaluation is an important strategy. Objective outcome evaluation via randomized trials is also commonly regarded as the โgold standardโ in establishing causal relationships. Despite its credibility, there are several problems of randomized trials. First, time is needed as longitudinal data are normally collected. Second, it is expensive because much manpower is involved in data collection over time. Third, attrition is a common problem in longitudinal evaluation studies. Nevertheless, experimental approach is still a widely used strategy to examine effectiveness of intervention programs. In the field of prevention, there are many examples of programs using trials to evaluate the treatment effects [27โ30].
The purpose of this paper is to examine the effectiveness of a positive youth development program (Project P.A.T.H.S.) in Hong Kong by using a validated measure of positive youth developmentโthe Chinese Positive Youth Development Scale. This is the first known scientific study that adopted a randomized group trial design using longitudinal data to evaluate a positive youth development program in the Chinese context. Consistent with previous longitudinal results [5โ7, 31], participants from the experimental group generally performed better than those from the control group in terms of different positive youth development indicators. The experimental subjects, as compared to their control counterparts, had a more stable rate of growth among the three subscales of the CPYDS (i.e., moral competence, behavioral competence, and the CPYDS-2). These findings were further supported based on the experimental subjects who found the program to be beneficial to their development. The treatment effects on the linear, quadratic, and cubic slopes indicating the program successfully altered the trajectory of psychological development among adolescents [32]. It is noteworthy that this effect sustained even after the completion of the program for two years (i.e., Waves 7 and 8).
Recent analyses based on this cohort revealed that the positive impact of the Project P.A.T.H.S. on reducing problem and risk behaviors [9]. Given the paucity of positive youth development programs using strong experimental or quasiexperimental designs in Asian countries [4], this study provides the strongest evidence to date regarding the beneficial effects of the Project P.A.T.H.S. on improving adolescent psychological development. In conjunction with the existing findings [3, 33โ37], the present study strongly suggest that the Project P.A.T.H.S. is able to promote positive development and reduce adolescent problem behavior in Chinese adolescents in Hong Kong.
In the present study, significant results that were more pronounced among those who perceived the program to be effective deserves some discussion. First, the findings suggest that positive youth development programs might not work in a โstimulus-responseโ manner and the cognitive appraisal of the participants is in fact important in determining the program outcome. As this factor is not properly addressed in the literature, it is recommended that further study should be carried out to examine how the subjective appraisal of the program participants might affect program outcomes through alteration of their motivation to join and participate. Second, the findings suggest that it is important to attend to the subjective outcomes perceived by the participants. As such, how to promote a sense of success and program ownership may help to promote program effectiveness in the long run.
Another significance of the study is the use of individual growth curve modeling to evaluate the impact of a positive youth program in a large sample across a five-year period, which is scarce in positive youth program literature. Researchers noted the needs of using advanced modeling techniques in longitudinal research [14, 38, 39]. Clearly, our study appears to be a positive response to this request.
There are several limitations in the present study. First, the data were based on self-report measures. Future research should evaluate the program by collecting longitudinal information from multiple methods (e.g., group interviews, diaries, and process evaluation) and sources (e.g., teachers, social workers, parents). Second, future research should examine the longitudinal effect of the positive youth development qualities on promoting psychological well-being (e.g., life satisfaction) and reducing problem behaviors (e.g., substance use, deliberate self-harm). This is supported by recent longitudinal findings which showed that positive youth development programs such as the Project P.A.T.H.S. can help to promote youth development and reduce their negative behavior among Hong Kong adolescents [8โ10]. Despite the above limitations, the present study demonstrates the effectiveness of the Project P.A.T.H.S. in promoting positive youth development among Hong Kong adolescents. Basically, the study underscores the importance of designing positive youth programs for adolescents.
Acknowledgment
The preparation for this paper and the Project P.A.T.H.S. were financially supported by The Hong Kong Jockey Club Charities Trust.