Evaluation of the Training Program of the Project P.A.T.H.S.: Findings Based on the Perspective of the Participants from Different Cohorts

Abstract

Subjective outcome evaluation findings based on the perspective of the participants participating in a 3-day training program of the Project P.A.T.H.S. are reported in this paper. The findings were based on the data collected from the training workshops conducted between 2005 and 2009 . Results showed that the respondents had good and positive perceptions of the training program and found it very valuable, particularly with respect to training instructors and familiarization with the project. Besides, the training participants were able to acquire attitude, knowledge and skills that are conducive to the successful implementation of the program. Based on the subjective outcome evaluation findings, it is concluded that the training program was effective in helping the participants to acquire the necessary knowledge, attitudes and skills in implementing the program.

1. Introduction

To promote positive and holistic development of adolescents in Hong Kong, The Hong Kong Jockey Club Charities Trust provided an earmarked grant of HK$ 750 million (HK$ 400 M for the first cycle and HK$ 350 M for the second cycle) to initiate and support the development, implementation, and evaluation of a positive youth development program entitled “P.A.T.H.S. to Adulthood: A Jockey Club Youth Enhancement Scheme” for junior secondary school students. The Project P.A.T.H.S. is a ground-breaking, indigenously developed, multiyear positive youth development program in Chinese context. There are two tiers of the programs, with the aims to promote holistic development of adolescents by providing opportunities and recognition for them to develop competence and skills which are conducive to positive youth development, promoting bonding with others, and holding healthy beliefs and clear values. The Tier 1 Program is a multi-year universal program designed for all Secondary 1 to Secondary 3 students based on 15 positive youth development constructs identified from the existing successful positive youth development programs [1, 2], while the Tier 2 Program is designed for students with greater psychosocial needs. With the support of the education and social welfare sectors, almost half of all the secondary schools in Hong Kong have joined the Project P.A.T.H.S. since its launch in 2005. From 2005 to 2009, there were altogether 223,101 students participating in the Tier 1 Program. Curricula for Secondary 1, Secondary 2, and Secondary 3 students were developed and tailored for Chinese adolescents by the research team comprising academics from five universities in Hong Kong.

1.1. Training Program of the Project P.A.T.H.S

In different fields, substantial time, money, man power, and energy are usually invested in in-service training programs. It is generally believed that investments in in-service training are associated with individual and organization benefits. In the same vein, in-service teacher training is commonly seen as one of the means to help teachers continue to acquire and enhance their professional knowledge and skills [3–6]. In addition, professional development can help enhance teacher performance and improve the learning and achievement of the students [3, 7, 8]. Guskey [9] pointed out that “high-quality professional development is a central component in nearly every modern proposal for improving education” (page 381).

In Hong Kong, there are various in-service training programs offered for teachers and social workers, which are mainly organized and conducted by the Education Bureau, Social Welfare Department, and other nongovernmental organizations. Like many practitioners in other fields, teachers and social workers may be unenthusiastic to adopt new programs or practices (as they may need to use a very different instructional approach during the implementation of new program) unless they feel secure and confident that they can make them work. Frontline experiences show that teachers’ confrontation might hamper the implementation of a new program. It is believed that the provision of pre-implementation training can increase the confidence of the program implementers. Veenman et al. [6] suggested that in-service training can serve three main purposes: “(1) to stimulate the professional competence and development of teachers; (2) to improve school practice; and (3) to implement political agreed-upon innovations in schools” (page 303). To ensure a smooth and effective implementation of the Tier 1 Program of the Project P.A.T.H.S., pre-implementation training for the program implementers is of great importance and the investment of in-service training is considered to be justified.

Systematic and training programs were designed and offered to all participating schools in the P.A.T.H.S. Project. The program implementers (teachers and/or social workers) involved in the Project were invited to participate in a 3-day training workshop, consisting of 20 hours of training for each grade (Secondary 1 to Secondary 3). Each training workshop provided 12 sessions of training, which were held in 3 days within the same week. The 3-level progressive training program has six general objectives which to help the potential program implementers: (a) to understand the nature of adolescent development and the related issues and to cultivate a positive attitude to adolescent development; (b) to understand the nature of positive youth development, including its basic concepts, related programs, and research; (c) to familiarize themselves with the nature of Project P.A.T.H.S., including its basic philosophy, design, implementation, and evaluation; (d) to understand the content of the Tier 1 Program, including the core program and elective program; (e) to acquire the attitude, knowledge and skills that are conducive to the successful implementation of the Tier 1 Program; (f) to establish a self-help support network among the program participants [10]. The six objectives were addressed throughout the training workshops. Through interactive training, potential program implementers can recognize how to deliver the programs effectively. It was also expected that participants’ motivation and sense of ownership of the program would be developed and hence enhancing successful program implementation.

According to Shek and Chak [10], the training programs of the Project P.A.T.H.S. have several unique features. First, progressive training programs are tailored for the program implementers of Secondary 1 to Secondary 3 levels, stage by stage from introductory level (Secondary 1) to advanced level (Secondary 3). Second, a 3-day program at each grade is designed. The first day (Day 1) gives the background of the project, which includes perspectives on adolescent development, positive youth development, project design, program implementation issues, and evaluation mechanisms. Day 2 and Day 3 of the training focus on discussing the related teaching methodologies and units in the Tier 1 Program with reference to different positive youth development constructs. In addition, ways to promote worker efficacy and reflection are included. Third, the training was experiential in nature with teachers actively engaged in all the lessons of the curriculum as participants, thereby demonstrating participatory and facilitative learning. Fourth, open discussion and interaction is emphasized in the training programs. Fifth, conscious effort is made to promote the passion and involvement of the potential program implementers. Finally, reflective learning is strongly emphasized in the training programs.

2. Evaluation of the Training Program

To appraise and determine the effectiveness of the training programs, program evaluation is indispensable. Scriven [11] defined evaluation as “the process of determining the merit, worth, and value of things and evaluations are the products of that process” (page 1). Guskey [12] emphasized the necessity of the existence of evaluation in professional development, especially in education sector. Although different in-service training programs are regularly provided for teachers and social workers, most of the developers of in-service training did not pay much attention to evaluation and thus very few programs have been empirically evaluated or documented [13]. Without such evaluation, it is not easy to determine if the training is beneficial to the training participants and their students or clients. At the same time, as published work on positive youth development training programs in Chinese communities is virtually, it is very important to document the details of the training programs. Against this background, this paper documents the evaluation findings related to 121 training workshops carried out during the Experiential Implementation Phase and Full Implementation Phase of the Project P.A.T.H.S. for the Secondary 1, Secondary 2, and Secondary 3 levels from school year 2005 to 2009.

Definitely, the training participants must be engaged in the evaluation process and their voices should be heard. As stated by the Joint Committee on Standards for Educational Evaluation [14], stakeholders should be identified (Standard U1) and the evaluation should take into account the different views of the stakeholders (Standard F2). According to utilization-focused evaluation proposed by Patton [15], significant stakeholders should be included in the evaluation process. When evaluating training effectiveness, numerous methods may be utilized. Obviously, one of the most common and direct ways to evaluate and determine the effectiveness of a training program is to understand the subjective views and reaction of the program participants (i.e., subjective outcome evaluation or client satisfaction approach). Regarding the outcome evaluation of training program, the classic four-level training evaluation model proposed by Kirkpatrick [16, 17] is dominant and well denoted in the literature of the organizational training evaluation. This model provides a straightforward guideline and reduces the measurement demands for training evaluation. The four levels of evaluation criteria are reaction (Level I), learning (Level II), behavior (Level III), and results (Level IV) criteria. This approach emphasizes the progresses from Level I to Level IV, and each level builds on those that come before. The evaluating criteria for each level are outlined as follows.Level I: reaction is a measure of participants’ perceptions to the training. It is basically a measure of client satisfaction.Level II: learning is a measure of what the participants have learnt and gained from the training.Level III: transfer is a measure of changes in their behavior when they return to the job after the training program. It also refers to the transfer of training, which measures knowledge and skills gained in the training that are applied on the job.Level IV: results are a measure of the final outcomes and the change in productivity that occur owing to the contribution of the training.

Although a number of criticism [18–20] and modifications to the model [12, 21, 22] have been suggested and recommended, Kirkpatrick’s four-level model of training evaluation continues to be the most prominent and influential among the existing training evaluation frameworks. In the present paper, subjective outcome evaluation findings are reported to evaluate the training program in the P.A.T.H.S. Project. With reference to Kirkpatrick’s model as a discussion framework, the first three levels (i.e., criteria of reaction, learning, and behavior) are highlighted. In the model of Kirkpatrick, the reaction evaluation (Level I) and the learning evaluation (Level II) try to measure how the participants react to the training program and what they have learnt and gained, which is also known as subjective outcome evaluation or client satisfaction evaluation. This type of evaluation describes the subjective views of the training participants, which comprise strong parallels with Kirkpatrick’s model to measure different training results. Participants’ perceived changes in behavior (Level III) were also emphasized in the training programs. Though some comments have criticized that the four-level evaluation model is too simple, this model offers a good starting point as “framework” to formalize the criteria of various outcomes that should be evaluated for a training program.

3. Methods

3.1. Participants and Procedures

A total of 121 workshops were conducted between 2005 and 2009 (Table 1), with 4,778 participants. Among these 3-day workshops, there were 49 workshops for the Secondary 1, 41 workshops for the Secondary 2, and 31 workshops for the Secondary 3 programs. At the last session of each training workshop, all participants were invited to respond to a structured but anonymous questionnaire (subjective outcome evaluation was the primary evaluation strategy used in this paper). This questionnaire focuses on the perceptions of the participants towards the program content, activities format, program instructors, self-performance, and administrative arrangement. All participants responded to all items in the evaluation form in a self-administration format. Provisions were also made for open-ended responses to enable respondents to make comments of appreciation or provide suggestions on matters not covered by the close-ended questions with predefined answers. Adequate time was given for the participants to complete the questionnaire. After collecting the data, the training team of Project P.A.T.H.S. input the data into an EXCEL file, which is used to compute the frequencies and percentages associated with the different ratings for the items. From 2005 to 2009, a total of 4,167 questionnaires were completed and collected (2,070 for the Secondary 1 level, 1,343 for the Secondary 2 level, and 754 for the Secondary 3 level) (Table 2). The overall response rate was 87.2%.

Table 1 

Information and attendance statistics of the training program of Project P.A.T.H.S. from 2005 to 2009.

Table 2 

Information of the respondents.

3.2. Instruments

The 31 items of the questionnaire were used to assess the participants’ satisfaction with the training program, the instructors, as well as their views towards their own performance. There are several sections in the subjective outcome evaluation questionnaire outlined as follows:(1)participants’ basic demographic information,(2)participants’ perceptions of the training program, including the program objectives, design, activities format, and interaction among the participants (16 items),(3)participants’ perceptions of the instructors, including the understanding of the course, teaching skills, and professional attitude (5 items),(4)participants’ perceptions of their own performance, including involvement during program, application of their learning, and having confidence in the project implementation (4 items),(5)participants’ perceptions of the administrative arrangement, such as program enrolment, hospitality, venue, and facilities (6 items),(6)things that the participants appreciated most (open-ended question),(7)aspects of the program that require improvement (open-ended question).

4. Results

The questionnaire consisted of 31 items with a six-point Likert scale (1 strongly disagree to 6 strongly agree). Reliability analysis shows that the internal consistency of the whole scale was good (alpha 0.96 for the total scale). Besides, all subscales on the training program (16 items), instructors (5 items), participants’ own performance (4 items), and administrative arrangement (6 items) were also reliable. The alpha values, mean interitem correlation are presented in Table 3.

Table 3 

Mean and standard deviations among the variables by grade.

The mean number of participants per workshop was 39.5. The total number of evaluation questionnaires completed was 4,167 (Table 2). Among these respondents, 68.7% of them () were female and 31.2% were male (), with 5 participants not disclosing their gender. In addition, 72.2% of the participants () were teachers while 26.1% of the participants () were social workers. There were 66 participants from other disciplines such as teaching assistants or program workers. The mean years of the self-reported work experience were 10.5 years, from a minimum of 0 to a maximum of 45 years (, SD = 8.18).

Several observations can be highlighted from the quantitative findings based on the closed-ended questions (31 items). First, the participants generally had a very positive perception of the program contents and activities format of the training program (Table 4), including cultivation of participants’ positive attitude to adolescent development (97.1%, ), strengthening of the participants’ understanding of positive youth development (97.5%, ), encouragement of instructors to do their best (97.5%, ), promotion of the participants’ understanding of the Project P.A.T.H.S. including its basic philosophy, design, implementation, and evaluation (97.5%, ), enhancement of participants’ understanding of the Tier 1 Program (97.3%, ), and strengthening of the participants’ understanding of the nature of adolescent development (96.1%, ). In particular, 97.3% of the respondents () cherished the peer-interaction amongst participants and 93.3% of the respondents () perceived that other participants were satisfied with the training program as well.

Table 4 

Summary of the views of the participants towards the contents and activities format of the training program.

Second, as indicated in Table 5, most of the participants (95.9%, ) perceived the instructors in a positive and encouraging manner (Table 5). About 98.7% () of the respondents thought that the instructors showed good and professional attitudes. 96.1% () of respondents perceived that the instructors had good mastery of the training curricula and that their teaching was clear and easy to understand (96.4%, ). Third, regarding the performance of the program participants during the training process, a high proportion of the respondents had positive evaluation of their own performance (97.1%, ) in the training program (Table 6). For instance, most of the participants perceived that they participated actively during discussion (95.6%, ). In addition, they reflected that they are willing to apply the specific skills and theories learnt from the training program (97.8%, ) and having confidence in program implementation after attending the training program (94.4%, ). Finally, as shown in Table 7, the participants had good evaluation of the administrative arrangement, most of them were satisfied with the administrative arrangement provided (98.7%, ) and the reception provided by the training team (98.8%, ). The participants also appreciated the workshop assigned (96.9%, ) and the venue facilities (96.3%, ).

Table 5 

Summary of the views of participants towards program instructors.

Table 6 

Summary of the views of participants towards themselves.

Table 7 

Summary of the views of participants towards administrative arrangement.

5. Discussion

The subjective outcome evaluation findings based on the responses of the potential program implementers in the Project P.A.T.H.S. are presented in this paper. According to the evaluation results, the current outcome evidently indicated that the training program had been successfully implemented and the program outcomes were very encouraging. With reference to Kirkpatrick’s four-level model, the present quantitative findings showed that the training programs of the project P.A.T.H.S. generally generated positive reactions (Level I), enhanced learning (Level II), and desired behavioral changes (Level III).

Several observations can be highlighted from the present findings. First, in relation to reaction data (Level I), favorable reactions were found. The training program was well received by the training participants. A large majority of participants reported that the program met their expectations. For instance, the overwhelming majority were satisfied with the training program, its content and format, training instructors, and the overall administration arrangement. As such, it is considered that the first four objectives of the training program have been achieved. In fact, the judgments about the worth and merit of the training program are very critical. If the program implementers were not satisfied with the training experience, they might not buy-in the Project P.A.T.H.S. and would not use what they have learnt in their teaching. They may even advise their colleagues not to buy-in the project and join the training program. Consequently, the well-designed training programs contributed and facilitated the development of positive attitudes among the program implementers towards the P.A.T.H.S. Project.

With reference to the domain of learning (Level II), the results provide reliable and constructive evidence. From the perspectives of the participants, there is no doubt that the training was tremendously valuable. For example, most of the participants indicated that the program provided sufficient information about the Project, which can help them cultivate their understanding of the nature of youth and positive attitude to adolescent development. In addition, almost all the respondents perceived that the training program strengthened their understanding of positive youth development and Project P.A.T.H.S., including its philosophy, program design, implementation and research. All these positive feedbacks suggest that learning effects were found in the training program. It is believed that the participants could gain knowledge or increase their awareness of the new program implementation. The P.A.T.H.S. training therefore seems to have fulfilled its fifth objectives. In particular, the findings revealed that the knowledgeable and skillful facilitation of the training instructors was highly appreciated and valued. As such, the program implementers could gain an understanding of the humanistic approach to teaching and actual implementation of the Project P.A.T.H.S. though the train-the-trainer workshops. Furthermore, a large majority appreciated the peer-interaction amongst participants. The training workshops actually help them to establish a self-help support network; the interactions among program participants and with the instructor are crucial for social learning. It is believed that the training of Project P.A.T.H.S. provided a supportive environment for the participants to grow and to learn from one another through interaction with other members. This addressed the sixth objective of the training program.

Concerning the behavior and performance change in teacher practice (Level III), the impact should be evaluated after a period of time to ensure the change has occurred. Based on the quantitative findings, it is shown that there were positive influences on the beliefs and attitudes of the participants, and the results suggest that the confidence of the program implementers was boosted. For instance, a high proportion of participants indicated that they are willing to apply the specific skills and theories learnt from the training program. The positive intention of integrating newly learnt knowledge and skills into their practice suggests the support of training transfer and behavior change, and thus students will then benefit from teacher practices. Of course, post-training and implementation research in this context is important.

As indicated in Table 4, many participants agreed that the training program had promoted self-reflection and they expressed that they had confidence in future program implementation. Such reflection could help improve instructional effectiveness and bridge the gap between adopted new philosophy of positive youth development and usual practice. There is a close relationship between self-reflection of teachers and their abilities to integrate theory with practice. It is noteworthy that if program implementers feel confident in implementing a new and unfamiliar program, their resistance will be diminished accordingly, thus empowering program implementation positively. Hargreaves and Fullan [23] argued that teachers’ behaviors and beliefs were closely bound together. When teachers were empowered, they would become more involved and satisfied with their job, which in turn has a positive impact on student outcomes.

This study was aimed to evaluate the training program of the Project P.A.T.H.S. by using subjective outcome evaluation based on the perspective of the training participants. On the whole, the training program can be regarded as successfully implemented, and the training objectives were also well achieved. It also replicated the previous subjective outcome evaluation findings of the training program of the Experimental Implementation Phase [24–28]. Based on the principle of triangulation and utilization-focused evaluation [14], the present findings are consistent with the previous research findings which showed that most of the stakeholders had favorable perceptions of the training program.

There are several strengths of this study. First, this study investigated different aspects of subjective outcome evaluation, including views of the participants towards the training program, training instructors, perceived effectiveness, and overall satisfaction, and all these scales were found to be reliable. Second, a respectable sample size was used in the study. Actually, it is noteworthy that there are very few scientific studies on the training program of positive youth development programs using such a large sample in the international context. Third, this study is one of the few empirical studies on the pre-implementation training program for the program implementers in a positive youth development program in Chinese context. Therefore, the present study provides a significant contribution to the literature.

However, there are three limitations of this study. First, as only the first three levels of Kirkpatrick’s model are discussed in this paper, the discussion is limited. In particular, the use of subjective outcome evaluation findings as support for Kirkpatrick’s model should proceed with caution. Second, there is an alternative explanation for the present positive outcomes. Given the demand characteristic (i.e., group pressure or they consciously act in a favorable manner), the participants might give more positive evaluation or tend to focus on the positive side of the training. Nevertheless, this alternative explanation could be partially dismissed since all the participants were professionals and they were invited to reflect their views in a frank way with a serious manner before completing the questionnaire. Moreover, the questionnaires were anonymous in nature. Third, since the present findings are only based on quantitative data, collection of qualitative data with integration of the quantitative findings can give a comprehensive picture about the impact of the training programs. Despite these limitations, the present findings are pioneering addition to the Chinese database on positive youth development. Research studies have consistently showed that the Project P.A.T.H.S. was effective in promoting holistic development in Chinese adolescents [29–37]. With hindsight, it may be conjectured that the positive evaluation findings are a result of the quality training program of the project.

Acknowledgment

The preparation for this paper and the Project P.A.T.H.S. were financially supported by The Hong Kong Jockey Club Charities Trust.