Abstract
Subjective outcome evaluation findings based on the perspective of the participants of the Project P.A.T.H.S. (Positive Adolescent Training through Holistic Social Programmes) in nine datasets collected from 2005 to 2009 ( program participants) were examined in this paper. Based on the consolidated data with schools as units, results showed that the participants generally had positive perceptions of the program, implementers, and benefits of the program. More than four-fifths of the participants regarded the program as beneficial to their holistic development. Multiple regression analysis revealed that the perceived qualities of the program and the program implementers predicted perceived effectiveness of the program. Based on the subjective outcome evaluation findings, the present study provides support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong.
1. Introduction
In his review of adolescent developmental issues in Hong Kong, Shek [1] drew several conclusions based on the available statistics and research findings. First, the adolescent substance abuse problem was part of a changing scene in Hong Kong. Second, although the overall youth crime trend was relatively stable in the past decade, the crimes of shoplifting and stealing deserved our concern. Third, the adolescent mental health problem was a growing issue. Fourth, adolescent unhealthy life styles, such as smoking, early sex, and moral confusion, were issues of concern. Fifth, adolescents experiencing economic disadvantage were a growing problem in Hong Kong, with one-quarter of children and adolescents experiencing economic disadvantage. Sixth, unemployed and nonengaged youth were emerging problems in the past 2 decades. Seventh, family and parenting problems in families with adolescents deserved our attention, and the Social Development Index showed that there was a gradual drop in family solidarity in the past decade. In the past few years, adolescent substance abuse has appeared to deteriorate, and it is a topic attracting much public attention in Hong Kong [2, 3]. With reference to these adolescent developmental problems, one important question is how we can promote holistic development in young people in Hong Kong.
To promote holistic development among adolescents in Hong Kong, The Hong Kong Jockey Club Charities Trust approved HK$400 million to launch a project entitled “P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme” based on the perspective of positive youth development. The word “P.A.T.H.S.” denotes Positive Adolescent Training through Holistic Social Programmes. There are two tiers of programs (Tier 1 and Tier 2 Programs) in the P.A.T.H.S. Project. Whereas the Tier 2 Program is a selective program for students with greater psychosocial needs, the Tier 1 Program is a universal positive youth development program in which students in secondary 1 to 3 participate, normally, with 20 h of training in the school year at each grade, involving 40 teaching units that have been developed with reference to 15 positive youth development constructs [4]. These constructs include promotion of bonding, cultivation of resilience, promotion of social competence, promotion of emotional competence, promotion of cognitive competence, promotion of behavioral competence, promotion of moral competence, cultivation of self-determination, promotion of spirituality, development of self-efficacy, development of a clear and positive identity, promotion of beliefs in the future, provision of recognition for positive behavior, provision of opportunities for prosocial involvement, and fostering prosocial norms. Because of the overwhelming success of the program, the project has been extended for another cycle, from 2009 to 2012, with an additional earmarked grant of HK$350 million.
There were two implementation phases in the original phase of the project—the experimental implementation phase and the full implementation phase. For the experimental implementation phase (January 2006 to August 2008), 52 secondary schools participated in the project with the objectives of accumulating experience in program implementation and familiarizing the front-line workers with the program design and philosophy. In the 2006/2007 school year, the programs were implemented on a full scale at the secondary 1 level. In the 2007/2008 school year, the programs were implemented at the secondary 1 and 2 levels. In the 2008/2009 school year, the programs were implemented at the secondary 1, 2, and 3 levels. The experimental and full implementation phases for the first cycle were successfully completed [5].
To provide a comprehensive picture pertaining to the effectiveness of the project, a wide range of evaluation strategies were employed to examine the program effect, including objective outcome evaluation utilizing a randomized group trial; subjective outcome evaluation based on quantitative and qualitative data collected from the program participants and instructors; qualitative evaluation based on focus groups involving students and instructors; in-depth interviews with program implementers; student products, such as weekly diaries; process evaluation involving systematic observations of delivery of the program and interim evaluation. The available evaluation findings consistently provide strong evidence that the Project P.A.T.H.S. has a beneficial influence on students [6–9].
To examine the perceptions of the program participants concerning the effectiveness of the project, subjective outcome evaluation or the client satisfaction approach was used. In human services, the importance of involving service users or program participants in evaluation is advocated, and thus subjective outcome evaluation becomes popularly used to capture the viewpoints of the participants. To capture the viewpoint of the participants, client satisfaction surveys are commonly used as feedback for transforming services, to meet the users’ needs for planning and administration purposes, or simply as an indicator of program effectiveness from the participants’ perspective for research purposes. Although there are many criticisms of this approach, the client satisfaction approach is widely used in different service settings. As pointed out by Royse [10], “despite the generally positive bias and the problems associated with collecting representative samples of clients, there is much to recommend client satisfaction studies as one means of evaluating a program. Because professionals do not experience the agency in the same way as the clients, it is important to ask clients to share their experiences” (pp. 264-265).
Subjective outcome evaluation is a popular approach employed by different professionals in different fields, such as education, social work, psychology, medicine, and allied health professions. The commonly used method develops closed-ended rating scale items to quantify client satisfaction. For example, standardized rating scales, such as the medical interview satisfaction scale, consumer satisfaction questionnaire, and client satisfaction questionnaire, were developed to gauge client satisfaction and perceived helpfulness of the program. In fact, it is commonly argued that, with the use of valid and reliable measures of the perceptions of the program participants, subjective outcome evaluation can yield objective pictures about program evaluation.
Previous studies showed that roughly four-fifths of the program participants generally had positive perceptions of the program, instructors, and benefits of the P.A.T.H.S. Project. In addition, the findings are fairly stable in different cohorts of students in the experimental and full implementation phases [11–13]. Furthermore, program content and program instructors were found to be significant predictors of perceived benefits of the program. As the Project P.A.T.H.S. was implemented in different cohorts of students from 2005 to 2009, it would be illuminating to aggregate findings in different cohorts to form an overall picture regarding the satisfaction of the participants. With data collected from large samples over time, a more stable picture of the subjective outcome evaluation findings can be generated. Against this background, the present paper attempts to describe the profile of subjective outcome evaluation findings based on the perspective of the participants. In addition, predictors of the perceived effectiveness of the program were also examined in this study involving aggregation of different datasets.
2. Methods
2.1. Participants and Procedures
From 2005 to 2009, the total number of schools that participated in the Project P.A.T.H.S. was 244, with 669 schools in the secondary 1 level, 443 in the secondary 2 level, and 215 in the secondary 3 level (Table 1). Altogether, 223,101 students participated in the Tier 1 Program in these 5 years. In these three grades, the mean number of students per school was 167.28 (range: 5–280), with an average of 4.61 classes per school (range: 1–8). Among them, 46.27% of the respondent schools adopted the full program (i.e., 20 h program involving 40 units), whereas 53.73% of the respondent schools adopted the core program (i.e., 10 h program involving 20 units). The mean number of sessions used to implement the program was 22.77 (range: 3–66). While 51.54% of the respondent schools incorporated the program into the formal curriculum (e.g., liberal studies, life education), 48.46% used other modes (e.g., form teachers’ periods and other combinations) to implement the program.
After completing the Tier 1 Program, the students were invited to respond to a Subjective Outcome Evaluation Form for Students (Form A) developed by the first author. From 2005 to 2009, a total of 206,313 questionnaires were completed (104,717 for the secondary 1 level, 69,334 for the secondary 2 level, and 32,262 for the secondary 3 level). The overall response rate was 92.48%. To facilitate the program evaluation, the research team developed an evaluation manual with standardized instructions for collecting the subjective outcome evaluation data. In addition, adequate training was provided to the implementers during the 20 h training workshops on how to collect and analyze the data collected by Form A.
On the day when the evaluation data were collected, the purpose of the evaluation was mentioned and the confidentiality of the data collected was repeatedly emphasized to all of the respondents. The respondents were asked to indicate if they did not want to respond to the evaluation questionnaire (i.e., “passive” informed consent was obtained). All respondents responded to all scales in the evaluation form in a self-administration format. Adequate time was provided for the respondents to complete the questionnaire.
2.2. Instruments
The Subjective Outcome Evaluation Form (Form A) [11–13] was used to measure the program participants’ perceptions of the Tier 1 Program. Broadly speaking, there are several parts in this evaluation form as follows:(i)participants’ perceptions of the program, such as program objectives, design, classroom atmosphere, interaction among the students, and the respondents’ participation during class (10 items)(ii)participants’ perceptions of the workers, such as the preparation of the instructor, professional attitude, involvement, and interaction with the students (10 items)(iii)participants’ perceptions of the effectiveness of the program, such as promotion of different psychosocial competencies, resilience, and overall personal development (16 items)(iv)the extent to which the participants would recommend the program to other people with similar needs (1 item)(v)the extent to which the participants would join similar programs in the future (1 item)(vi)overall satisfaction with the program (1 item)(vii)things that the participants learned from the program (open-ended question)(viii)things that the participants appreciated most (open-ended question)(ix)opinion about the instructor(s) (open-ended question)(x)areas that require improvement (open-ended question).
For the quantitative data, the implementers collecting the data were requested to input the data into an EXCEL file developed by the research team that would automatically compute the frequencies and percentages associated with the different ratings for an item. When the schools submitted the reports, they were also requested to submit the soft copy of the consolidated datasheets. After receiving the consolidated data by the funding body, the data were aggregated in order to “reconstruct” the overall profile based on the subjective outcome evaluation data by the research team.
2.3. Data Analyses
Percentage findings were examined using descriptive statistics. A composite measure of each domain (i.e., perceived qualities of program content, perceived qualities of program implementers, and perceived program effectiveness) was created based on the total scores of each domain divided by the number of items. Pearson’s correlation analysis was used to examine if the program content and program implementers were related to the program effectiveness. Multiple regression analysis was performed to compare which factor would predict the program effectiveness. All analyses were performed by using the Statistical Package for Social Sciences version 17.0.
3. Results
The quantitative findings based on the closed-ended questions are presented in this paper. Several observations can be highlighted from the findings. First, the participants generally had positive perceptions of the program (Table 2), including clear objectives of the curriculum (83.50%), much interaction among students (81.84%), and well-planned teaching activities (81.76%). Second, a high proportion of the participants had a positive evaluation of implementers’ performance (Table 3). For example, the participants thought that the implementers were very involved (88.63%), ready to help them when needed (88.22%), and encouraged them to participate in activities (88.12%). Third, as shown in Table 4, many participants perceived that the program promoted their development, including moral competence (84.05%), compassion for others (81.75%), social competence (82.11%), and overall development (83.24%). Fourth, 77% of the participants would recommend the program to students with similar needs. Fifth, 65.37% of the participants expressed that they would participate in similar courses again in the future. Finally, 84.48% of the respondents were satisfied with the program on the whole (Table 5).
Reliability analysis with the schools as the unit of analyses showed that Form A was internally consistent (Table 6): 10 items related to the program (α = 0.98), 10 items related to the implementer (α = 0.99), 16 items related to the benefits (α = 1.00), and 36 items measuring overall program effectiveness (α = 0.99). Results of correlation analyses showed that both program content (, ) and program implementers (, ) were strongly associated with program effectiveness (Table 7).
Table 8 presents multiple regression analysis results. Higher positive views toward the program and program implementers were associated with higher program effectiveness (). Further analyses showed that program content (β = 0.75) was a significantly stronger predictor than program implementers (β = 0.24). This model explained 95% of the variance toward the prediction of program effectiveness. Interestingly, the above relationships and the amount of variance were consistent across grade levels.
4. Discussion
The purpose of this study was to evaluate the Tier 1 Program of the Project P.A.T.H.S. via the subjective outcome evaluation approach based on the perspective of the program participants using the data collected in the experimental and full implementation phases (2005–2009) of the project. There are several characteristics of this study. First, a large sample of schools (more than 200 schools per grade) and students (, 313) were involved. Second, different datasets collected at different points of time were analyzed in this study. Third, responses of students in different grades were collected. Fourth, this is the first known scientific study of the subjective outcome evaluation of a positive youth development program based on different cohorts in China. Finally, this is also the first study of subjective outcome evaluation based on such a large sample of participants in the global context.
Generally speaking, the quantitative findings showed that a high proportion of the respondents had positive perceptions of the program and the workers; roughly four-fifths of the respondents regarded the program as helpful to them. The findings basically replicated those findings reported previously based on the perspective of the program participants and they are also consistent with those based on the perspective of the program implementers. In fact, an examination of the percentages of responses to different items revealed that the figures were very similar across different studies. In conjunction with findings based on other evaluation strategies, the present integrative evaluation study showed that the Tier 1 Program of the Project P.A.T.H.S. was well received by the program participants, and over four-fifths of them were of the view that the program was beneficial to their development.
There are several contributions of the present study. First, in view of the lack of positive youth development programs and related evaluation findings in the Chinese context, the present study is a pioneer study. Besides showing that Project P.A.T.H.S. is effective, it also demonstrates how subjective outcome evaluation based on a large sample size can be carried out. Second, the findings show that the subjective outcome measure is reliable. Because there are few validated measures in the Chinese culture [14, 15], the present study contributes to the assessment literature on psychosocial measures in the Chinese context.
Finally, findings on the predictors of subjective outcome evaluation are important because there are currently few conceptual models on the determinants of subjective outcomes. There has been some discussion in the literature on how the quality of a program can be enhanced by tailoring an appropriate program to suit the values and needs of target populations [16, 17]. For example, using the Youth Program Quality Assessment (YPQA) instrument, researchers found that the effect of program delivery qualities varied with the students’ ages [18]. In addition, the positive youth-oriented approach was found to be more beneficial for high-school students, while staff-oriented pedagogy was more appropriate for elementary school students [17, 18]. Unfortunately, although program components and their interactions with individual factors are important determinants of the effectiveness of youth programs, very few studies have examined the effect of different program components on perceived program effectiveness, especially in the Chinese context. The present findings fill an important gap in the formulation of theoretical models on the determinants of subjective outcome evaluation of positive youth development programs.
Although utilization of subjective outcome evaluation or the client satisfaction approach in evaluation has a long history in human services, there are arguments against the use of subjective outcome evaluation. For example, subjective outcome evaluation has been criticized as biased and unable to reflect the real behavioral changes in the program participants [19, 20]. Nevertheless, there are several features in this study that may be used to argue against such criticisms. First, a very big sample was used in this study, with 206,313 students in roughly half of the secondary schools in Hong Kong. Such a big sample size substantially enhances the generalizability of the research findings and their credibility. Second, different aspects of subjective outcome, including views on the program, worker, perceived effectiveness, and overall satisfaction, were covered in the study. The present findings also showed that the Form A rating items were reliable with reference to the sections and the whole scale. According to Royse [10], the lack of standardized assessment tools for conducting a client satisfaction survey also introduces biases for the client satisfaction approach. As such, he recommended the use of an assessment tool with known reliability and validity that would “eliminate many of the problems found in hastily designed questionnaires” (p. 265).
Third, because the findings reported in this paper were “reconstructed” based on the reports submitted anonymously by the participating schools, the possibility that the students reported in an over-cooperative manner was not high. Finally, previous research findings based on the same project have shown that subjective outcome evaluation findings actually converged with objective outcome evaluation findings [21, 22]. In view of the lack of research data in this view, such studies point to the value of collecting subjective outcome evaluation data. Of course, the use of schools as the units of analyses might mask individual differences involved. However, in view of the large number of schools involved, this is not a particularly acute problem.
Despite these limitations, the present findings suggest that the Tier 1 Program of the Project P.A.T.H.S. and its implementation were perceived in a positive manner by the program participants. In conjunction with other evaluation findings, the present study suggests that the Tier 1 Program of the Project P.A.T.H.S. was perceived to be beneficial to the development of the program participants. With reference to the gradual decline of parental control in the early teenage years of Chinese adolescents in Hong Kong, positive youth development programs such as the Project P.A.T.H.S. are important initiatives to promote psychosocial competencies in Chinese adolescents of Hong Kong [23]. Furthermore, although subjective outcome evaluation is a popular approach used in human services in the Western contexts [24–28], there are comparatively few published studies in the Chinese contexts, particularly in the area of positive youth development. As such, the present integrative study and the related studies can be regarded as groundbreaking in the field of positive youth development in different Chinese contexts.
Acknowledgments
The preparation for this paper and the Project P.A.T.H.S. were financially supported by The Hong Kong Jockey Club Charities Trust. The authorship is equally shared between the first and second authors.