Abstract
A total of 231 schools participated in the Project P.A.T.H.S. in 2009/2010 school year. After completion of the Tier 1 Program, subjective outcome evaluation data were collected from 3,259 program implementers. Based on the consolidated data with schools as units, results showed that participants had positive perceptions of the program, implementers, and benefits of the program. More than four-fifth of the implementers regarded the program as helpful to the program participants. Multiple regression analyses revealed that perceived qualities of the program and the program implementers predicted perceived effectiveness of the program. Similar to previous studies, compared to implementers’ perception about their performance, the perceived program content appeared to be a stronger predictor of program success. The present study provides additional support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong.
1. Introduction
How to prevent adolescent risk behavior, such as delinquency, drug abuse, unprotected sexual behavior, and school failure, has been a challenging issue for psychologists, educators, policy makers, and other helping professionals [1–3]. In recent years, the research paradigm that different adolescent risk behaviors are treated as separate and independent problems is changing. Instead, emphasis has been put on the interconnections among various risk behaviors and their shared risk, protective, and facilitative factors. Both theoretical models and empirical studies have supported one common predictor of a wide range of risk behaviors in youth—positive youth development or youth developmental assets [4]. Accordingly, numerous youth programs have been developed with a focus on promoting the development of core competences and adaptive features of adolescents, which can be generally subsumed under the category of positive youth development approach [5–7].
The approach of positive youth development has been widely adopted in designing programs for adolescents in the west [8]. However, such programs are rarely developed and carried out in Asian countries, especially different Chinese communities [9]. In view of this situation, Shek and researchers from five universities in Hong Kong developed a large-scale positive youth development program entitled the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) to promote healthy development in Hong Kong adolescents and to prevent various youth risk behaviors [10, 11]. Funded by the Hong Kong Jockey Club Trust Charities since 2005 (with funding of HK$400 million in the initial phase and HK$350 million in the extension phase), the project has been implemented in about half of the secondary schools in Hong Kong for consecutively seven years. There are two tiers of program in the project. Tier 1 Program is a universal positive youth development program provided for secondary 1 to 3 students in Hong Kong. Tier 2 Program takes a selective approach which aims at around one-fifth of the Tier 1 Program participants who have greater psychosocial needs.
As the Project P.A.T.H.S. has been implemented in a large scale in Hong Kong adolescents, one important question that must be asked is how effective the project is. To answer this question, systematic evaluation of the program is necessary. Since the launch of the program, numerous evaluation studies have been carried out, with the use of a variety of evaluative strategies, including objective outcome evaluation, subjective outcome evaluation, focus group interview, case studies, direct observation, and a longitudinal randomized group controlled trial [12, 13]. Findings based on these evaluation studies in the past seven years have generally shown positive program effects of the Project P.A.T.H.S. in promoting different competences and developmental assets and preventing various risk behaviors in the program participants [14–17]. For example, based on eight waves of data collected in five consecutive years, Shek and colleagues reported that students who had participated in the Project P.A.T.H.S. showed better developmental outcomes than did students in a randomized controlled group, in terms of both positive youth development indicators (e.g., resilience, moral competence, and prosocial involvement) and different risk behaviors such as substance abuse and delinquent behaviors [18, 19].
While objective outcome evaluation, particularly randomized controlled trial, is considered the “gold” standard for the assessment of program effectiveness, subjective outcome evaluation has several unique advantages in program evaluation [20–22]. First, as compared to objective outcome evaluation, subjective outcome evaluation provides a way to find out different stakeholders’ opinions and subjective experiences of the program. Second, subjective outcome evaluation offers immediate and important information about the implementation of a program before its effects on objective indicators can be observed. Third, subjective outcome evaluation is a more cost-effective evaluative method than objective outcome evaluation. Fourth, subjective outcome evaluation by program implementers contains valuable message about problems and difficulties encountered in program implementation which contribute to the improvement of the program in the future. In evaluating the Project P.A.T.H.S., subjective outcome evaluation was conducted in both program participants and program implementers to obtain a comprehensive picture about different stakeholder’s views towards the project [23].
Although very encouraging evaluation findings have been reported for the initial phase of the project, it is important to know whether similar positive findings could be found for the extension phase. Against this background, subjective outcome evaluation findings based on the perspectives of program workers who implemented the Tier 1 Program in the 2009/2010 school year were reported in this paper. In addition, instructors’ perceptions about the program, their own performance, and the effectiveness of the project were contrasted among different grade levels to learn about whether program workers at different grades have different views about the program. Previous findings suggested that instructors who taught the curriculum in the lower forms had more positive perceptions than did instructors teaching the program in the higher forms. As such, it was hypothesized that similar pattern regarding the grade effect on program implementers’ subjective evaluation would also be observed in the present sample. Besides, the relationships among program implementers’ views towards the program, perceptions about the instructor, and the overall effectiveness of the program were examined to gain a further understanding of critical factors that influence perceived program effectiveness by program workers. Based on prior findings, it was hypothesized that program implementers’ perceived program quality and their own performance would significantly predict their subjective evaluation on program effectiveness.
2. Methods
2.1. Participants and Procedures
A total of 231 schools joined the Project P.A.T.H.S. in the fourth year of the full implementation phase in the 2009/2010 school year (i.e., the first year in the extension phase), with 219, 185, and 173 schools in secondary 1, secondary 2 and secondary 3 levels, respectively. The mean number of students per school was 154.36 (ranged from 6 to 240 students), with an average of 4.50 classes per school (ranged from 1 to 12 classes). Among them, 32.24% of the respondent schools adopted the full program (i.e., 20-hour program involving 40 units) whereas 67.76% of the respondent schools adopted the core program (i.e., 10-hour program involving 20 units). The mean number of sessions used to implement the program was 28.54 (ranged from 2 to 48 sessions). While 47.31% of the respondent schools incorporated the program into the formal curriculum (e.g., Liberal Studies, Life Education), 52.69% used other modes (e.g., using form teacher’s periods and other combinations) to implement the program. The mean numbers of social workers and teachers implementing the program per school per form were 1.71 (ranged from 0 to 7) and 5.11 (ranged from 0 to 27), respectively.
After the Tier 1 Program was completed, the implementers were invited to respond to a Subjective Outcome Evaluation Form (Form B) developed by the first author [24]. In the school year 2009-2010, a total of 3,259 questionnaires were completed. The data collection was conducted after the completion of the program. To facilitate the program evaluation, the Research Team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data [24]. In addition, adequate training was provided to the implementers during the 20-hour training workshops on how to collect and analyze the data collected by Form B.
2.2. Instruments
The Subjective Outcome Evaluation Form (Form B) was used in the present study, including the following parts.(i)Program implementers’ perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondents’ participation during class (10 items).(ii)Program implementers’ perceptions of their own practice, including their understanding of the course, teaching skills, professional attitude, involvement, and interaction with the students (10 items).(iii)Implementers’ perceptions of the effectiveness of the program on students, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items).(iv)The extent to which the implementers would recommend the program to other students with similar needs (1 item).(v)The extent to which the implementers would teach similar programs in future (1 item).(vi)The extent to which the program implementation has helped the implementers’ professional growth (1 item).(vii)Things that the implementers obtained from the program (open-ended question).(viii)Things that the implementers appreciated most (open-ended question).(ix)Difficulties encountered (open-ended question).(x)Areas that require improvement (open-ended question).
For the quantitative data, the program workers who collected the data were requested to input the data in an EXCEL file developed by the Research Team which would automatically compute the frequencies and percentages associated with the different ratings for an item. When the schools submitted the hard copy of the reports, they were also requested to submit the soft copy of the consolidated data sheets. After receiving the consolidated data by the funding body, the research team aggregated the data to “reconstruct” the overall profile based on the subjective outcome evaluation data. It should be noted that although both qualitative and quantitative data were collected, the present paper only focused on the quantitative reports. Qualitative findings are to be reported elsewhere.
2.3. Data Analysis
Percentage data were examined using descriptive statistics. A composite measure of each factor (i.e., perceived qualities of program content, perceived qualities of program implementers, and perceived program effectiveness) was created based on the total scores of each scale divided by the number of items. Pearson correlation analysis was used to examine if the program content and program implementers were related to the program effectiveness. To compare program implementers’ evaluation across different grades, several one-way ANOVAs were conducted with the three subscale scores as the dependent variables and grade as the independent variable. Hierarchical linear regression analyses were further performed to examine the relationship between different aspects of implementers’ evaluation about the project and the program effectiveness. All analyses were performed by using the Statistical Package for Social Sciences Version 17.0.
3. Results
The quantitative findings based on the closed-ended questions are presented in this paper. Several observations can be highlighted from the findings. First, the participants generally had positive perceptions of the program (Table 1), including clear objectives of the curriculum (94.80%), well-planned teaching activities (90.51%), and very pleasant classroom atmosphere (87.88%). Second, a high proportion of the implementers had positive evaluation of their performance (Table 2). For example, 98.18% of the implementers perceived that they were ready to help their students; 97.43% of the implementers expressed that they cared for the students; 96.35% believed that they had good professional attitudes. Third, as shown in Table 3, many implementers perceived that the program promoted the development of students, including their resilience (91.84%), social competence (93.73%), life reflections (91.61%), and overall development (93.16%). Fourth, 89.73% of the implementers would recommend the program to students with similar needs. Fifth, 83.36% of the implementers expressed that they would teach similar courses again in the future. Finally, 84.37% of the respondents indicated that the program had contributed to their professional development.
Reliability analysis with the schools as the unit of analyses showed that Form B was internally consistent (Table 4): 10 items related to the program , 10 items related to the implementer , 16 items related to the benefits , and the overall 36 items measuring program effectiveness . Results of correlation analyses showed that both program content (, ) and program implementers (, ) were strongly associated with program effectiveness.
To examine differences in the perceived variables (i.e., program content, program implementers, and program effectiveness) across grade levels, several one-way ANOVAs were performed with the perceived variables as dependent variables and grade level (i.e., secondary 1 to 3) as independent variable. Significant results were only found in program content, , . Post hoc analysis using Tukey’s procedure with Bonferroni adjustment (i.e., ) revealed that significant difference was found between secondary 1 and secondary 3 participants (), with the secondary 1 program perceived to be relatively more favorable than the Secondary 3 Program.
Multiple regression analyses were performed on both the whole sample and the responses of students in different grades separately. Table 5 presents the findings. Overall, higher positive views towards the program and program implementers predicted higher perceived program effectiveness (). The prediction of program effectiveness was stronger for perceptions of program than for views towards implementers . The model explained 63% of the variance toward the prediction of program effectiveness. For participants in different grades, the pattern of relationships and the amount of variance in program effectiveness explained by the two predictors were very similar. While views towards program content consistently predicted program effectiveness across grades, the relationship between views towards implementers and program effectiveness was only significant for the analyses based on the secondary 2 participants.
4. Discussion
The present study investigated the subjective outcome evaluation by program workers who implemented the Tier 1 Program of the Project P.A.T.H.S. in the 2009/2010 academic year. The findings showed that program implementers generally held positive views towards the program and the instructors and perceived the program as effective to promote healthy development of the participants. Program implementers’ perceptions about the program and instructor significantly predicted their subjective evaluation about the program effectiveness, with views towards program content being a stronger predictor than views towards instructors. Moreover, these findings were held true for participants from different grade levels. There are three unique features of this study. First, the sample size was quite large. Actually, it is very rare to see such a large number of program implementers participated in outcome evaluation in the literature. Second, a validated measure of subjective outcome evaluation was used. Third, as there are few studies on the evaluation of positive youth development programs in general, particularly in Chinese people, the present study is an important addition to the literature.
Overall, more than 80% of the participated program implementers had positive evaluation about different aspects of the program content, including the good curriculum design, strong theoretical support, pleasant classroom atmosphere, and active participation of the students. In particular, more than 90% of the instructors agreed that the objectives of the curriculum were very clear and the activities were carefully planned. Explicit learning objectives with respect to the required skills and a variety of instructional activities to facilitate learning are two critical components in outcome-based education which embraces the notion that the learner is accountable for his or her own achievements and represents the most updated approach to nowadays education [25–28]. The present findings that these two items received the highest subjective evaluation from teachers suggest that the outcome-based approach has been well incorporated in the implementation of the Project P.A.T.H.S. Other opinions from teachers such as “there was much peer interaction amongst the students” and “on the whole students like this curriculum very much” provide further support for the successfulness of using this approach to deliver the Project P.A.T.H.S. in Hong Kong students.
Program implementers also viewed their own performance in teaching the program favorably, in terms of mastery and preparedness of the curriculum, teaching skills and attitudes towards the course and students, personal gains, interaction with students, and general evaluation of oneself as an instructor of the program. While self-fulfilling prophecy may explain the findings, it is noteworthy that this observation is consistent with previous findings that the students also perceived the instructors in a favorable light [12, 29], hence supporting the validity of the present finding.
With respect to the perceived effectiveness of the program, program implementers regarded the program as having promoted positive development in the participated students in multiple areas. For example, more than 90% of the instructors agreed that the project had enhanced students’ bonding with others, resilience in adverse conditions, social competence, ability to make sensible and wise choices, and overall development. Students who attended the program were evaluated as having more life reflections and self-awareness. These findings are consistent with previous results based on other evaluation methods regarding the effectiveness of the Project P.A.T.H.S., such as the objective outcome evaluation and the subjective evaluation by students [29, 30].
Program implementers’ subjective outcome evaluation was also compared among different grades. No significant grade differences were detected in program implementers’ views about their own performance and perceived effectiveness of the program, which suggests that program implementers from different grade levels had similar favorable views towards the instructor and the program effectiveness. However, it was found that program content was evaluated more positively by secondary 1 implementers than by secondary 3 implementers. Similar findings were also noted in previous studies. While the curriculum designed for different grades has different content including various activities and topics for discussion, the basic framework of the course that consists of eight core positive youth development constructs is the same across grade. Therefore, the secondary 1 program may be perceived as more fresh and attractive to teachers than secondary 3 program. Besides, students in junior grade may also show more interests and better involvement in the course than senior students who attended the program since they entered to the secondary school. This finding provides some insights for the curriculum design in the future. Perhaps more novel units and topics especially suitable for senior secondary students could be developed and incorporated into the curriculum. Despite the grade difference, program implementers in the secondary 3 grade still reported favorable views towards the curriculum, with more than three fourths of the participants having positive evaluation about different aspects of the program content, which suggests that the curriculum is generally well received by the instructors.
Results of regression analyses suggest that for the whole sample of students, both perceived program and instructors significantly predicted the perceived effectiveness of the program, supporting the critical roles of program quality and implementers in program success. However, when data in different grades were analyzed separately, while program workers’ subjective evaluation of the program quality consistently predicted perceived effectiveness of the program across grade, the effect of views about instructors’ performance was only significant for secondary 2 participants. Apparently, program worker’s evaluation about the program content appeared to be a stronger predictor than did their evaluation about instructors’ performance. Similar findings were reported in Shek et al.’s paper [31]. While a variety of factors at different ecological levels were found to affect the implementation of a program, high program quality has always been considered the first requisite to the success of the program [32, 33]. Without a good design of the curriculum in the very beginning, it is impossible that the program will produce desirable outcomes in its participants, even with excellent program staff, highly-motived students, and supportive administrative environment. In fact, it is likely that quality of program and quality of implementers interactively affect the effectiveness of a program. For example, good curriculum content often increases the interests and motivation of instructors to teach the course [34], and thus the instructors may spend more time in preparation, show more passion in their teaching, and deliver the content in a more effective way. Therefore, when the effects of program content were controlled, the prediction of program instructors’ performance on program effectiveness decreased. Another possibility is that this may be a statistical artifact as the range of scores for the evaluation of instructors was not wide. Future studies may focus on examining the interactive effects between program content and program implementers to identify more fundamental factors that determine program success. In addition, it is unclear why in the present study the evaluation of instructors only predicted program effectiveness for secondary 2 participants, but not for secondary 1 and 3 participants. This finding is inconsistent with previous report [31] and the literature in which the critical role of program implementers to program success is constantly highlighted [13, 21]. Obviously, replication study is needed. In particular, grade difference in the effect of program implementers’ performance on program effectiveness should be further explored.
There are several limitations of the present study that should be acknowledged. First, the data were collected in a self-reported manner, which may be biased by the implementers’ personal attitudes and perceptions towards the program. To reduce the potential bias, several measures were taken. First, program implementers responded to the questionnaire anonymously, and the confidentiality was repeatedly assured. Second, in the questionnaire, no threatening questions were asked that might elicit the respondents’ feelings of role conflict and social desirability. Third, participants were encouraged to candidly report their negative views or feelings in the survey, and open-ended questions were provided for the teachers to record their suggestions on how to improve the program. Despite of these measures, the present findings should be interpreted with cautions, and evaluative studies that use other approaches, such as objective outcome evaluation based on developmental indicators, program participants’ subjective evaluation, and process evaluation must be conducted for the purpose of triangulation. The second limitation of the present study is that only two general indicators of program quality and program implementers’ performance were used to predict overall effectiveness of the project, which makes it impossible to identify specific aspects that are particularly important for program success. Besides, different factors may increase/decrease the program effects in different areas. For example, good performance of the teacher may have particular effects in strengthening students’ bonding with teachers. Future studies may include different indicators of program content and implementers as well as program effectiveness in the prediction model. Thirdly, previous studies have revealed that school and organization characteristics influence program effectiveness and implementation quality [34–36]. These contextual factors should be considered in further research. Finally, as the present findings were “reconstructed” from the evaluation reports submitted by the agencies, the unit of analyses was schools, instead of individuals. Therefore, individual variations were lost in the process which may lower the power of statistical analyses. Despite of these limitations, the present study constitutes an important addition to the current literature about the effectiveness of the Project P.A.T.H.S. in promoting positive youth development in Hong Kong adolescents.
Acknowledgment
The preparation for this paper and the Project P.A.T.H.S. were financially supported by The Hong Kong Jockey Club Charities Trust.