Abstract
This paper aims to investigate the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. (positive adolescent training through holistic social programmes) based on eight datasets collected between 2005 and 2009. A total of 1,138 students who participated in the program were randomly invited (from the whole grade or in some classes) to write a piece of journal in the form of a weekly diary in order to reveal their perceptions and feelings regarding the program and the perceived benefits of the program. Based on an integration of findings from different databases, results showed that the respondents generally (1) had positive views on the program, (2) had positive views on the instructors, and (3) perceived that they had acquired competencies at the societal, school, familial, interpersonal, and personal levels after joining the program. Acknowledging the limitations of diaries, the present qualitative findings provide support for the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong.
1. Introduction
The purpose of this paper is to report evaluation findings of a positive youth development program in Hong Kong (Project P.A.T.H.S., i.e., positive adolescent training through holistic social programmes) based on weekly diaries written by students participating in the program. The personal diary is an important source of information to understand the social reality. For example, Anne Frank’s diary documents the thoughts, feelings, and experiences of a Jewish teenage girl who had fled and hid in Amsterdam with her family when Nazi forces conquered The Netherlands. In view of the invaluable information gathered from people’s diaries, diaries for analyzing human behavior, discovering history, or evaluating a program represent an attractive research approach. In the context of social sciences research, diaries are commonly used in nonexperimental research to collect data on human behavior. For example, Leigh [1] asked participants to record their time, quantity, and type of drinking and sexual activity over 10 weeks in order to examine the temporal relationships of alcohol consumption to sexual behavior. Behavioral diaries have also been used in behavioral modification programs [2]. There are studies to suggest the strengths and weaknesses of using daily diaries as a research strategy [3, 4], and the debate has continued in the literature [5, 6].
In the field of education, it is not uncommon to find that a teacher would keep reflective journals as a means to assess personal teaching performance, identify one’s pedagogical strengths and weaknesses, and evaluate the effectiveness of the changes implemented in teaching [7]. It is believed that a reflective journal “suggests question, identifies new areas to explore, reveals meaningful absences, and uncovers recurring patterns” ([8], page 16). Therefore, some studies utilized teachers’ reflective diaries as a kind of data collection method to investigate the essential components of effective teaching practice or pedagogy, and to examine ways of contributing to professional development [9].
Similarly, in contrast to the traditional learning model where students mainly take a passive role in receiving what their teachers’ transmit or reciting the information from books, students are perceived as active learners who construct their personal knowledge based on their own experiences in reflective learning models, where they are encouraged to be autonomous learners who reflect on their learning and to have more awareness on their learning processes, competencies, and limitations. Therefore, reflective observation is highlighted to be an indispensable process in translating concrete experiences to concepts and actions in the experiential learning process [10], in which keeping a diary is considered as an opportunity for personal reflection that, in turn, enhances students’ autonomy and develops an ownership of their learning process [11].
In order to guide students to become autonomous learners, students studying in human services programs were commonly asked to write diaries or reflective journals in different ways. First, some courses simply aimed to encourage students to be critical and reflective learners. For example, in a module on management education, students were asked to write eight diaries with predesigned topics throughout the course [12]. Second, some courses were targeted towards strengthening the students’ ability to integrate their course learning into actual field settings. For example, social work students were required to write reflective journals with predesigned questions at several stages of their field work, on top of other written assignments [13]. Third, students were guided to be determinant masters of their future. For example, students in a health-discipline course were asked to write a journal with no specific guidelines on their experiences of their practices in rural areas, so as to identify the positive and negative aspects of rural practices and develop their interest in working in the rural areas in the future [14]. Chaloner [15] affirmed that “using a personal journal implies that the learner will construct a memory of the learning experience and how it can be applied to the workplace by using a personal narrative. Through such an individualized and personal approach, it is hoped that the learner will be motivated enough to assume the autonomy that he/she is being offered” (page 22).
Besides understanding the subjective world views of the informants, a survey of the literature in the field of evaluation shows that there is an increasing number of studies using diaries as a tool for evaluation. Wagner [16] used student journals to evaluate the effectiveness of a course. Follick et al. [17] evaluated a daily activity diary for chronic pain patients. Travers [18] examined the use of a reflective diary methodology for exploring the lived experiences of stress and coping among 30 college students. Cohen et al. [19] examined the development and use of an online diary in the evaluation of a preventive care program that provided real-time communication between the evaluator and participants. Knowles and Tarrier [20] evaluated diary intervention on levels of anxiety and depression in a group of intensive care unit survivors and demonstrated the therapeutic effects of diaries.
The main purpose of this paper is to report evaluation findings of the Project P.A.T.H.S. using student diaries as an evaluation method. The project P.A.T.H.S. is a positive youth development program that attempts to promote holistic development among adolescents in Hong Kong, with an initial earmarked grant of HK$400 million by The Hong Kong Jockey Club Charities Trust. The Project is divided into two tiers of programs. The Tier 1 Program is a universal curriculum-based program for secondary 1 to 3 students in participating schools and the Tier 2 Program is a selective program for students with greater psychosocial needs. In contrast to mainstream approaches that focus on tackling youth developmental problems, the field of positive youth development focuses on enabling the talents, strengths, interests, and future potentials in children and adolescents. Roth et al. [21] pointed out that positive youth development programs are commonly regarded as “programs that provide opportunities and support to help youth gain the competencies and knowledge they need to meet the increasing challenges they will face as they mature” (page 423). Although many western programs have been developed to promote positive development in children and adolescents, not all of them are successful in promoting adolescent development. For example, Catalano et al. [22] reviewed 77 programs on positive youth development, and their review showed that only 25 programs were successful. Obviously, evaluation is an important issue to be considered when assessing the effectiveness of positive youth development programs.
To understand the program effects of the Project P.A.T.H.S., several complementary program evaluation strategies, including both quantitative and qualitative methodologies, have been adopted [23]. These included objective outcome evaluation, subjective outcome evaluation, process evaluation, interim evaluation, and qualitative evaluation based on focus groups, in-depth interviews, and case studies. Through the adoption of different evaluation strategies, there is multiple evidence supporting the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. based on different types of data collected from different participants utilizing different methods. For example, both quantitative findings [24, 25] and qualitative findings [26, 27] based on different studies consistently suggest that the Project P.A.T.H.S. is helpful in promoting the psychosocial competencies of the program participants.
To further understand the perceived program effects of the Tier 1 Program, evaluation findings based on weekly diaries written by students are reported in this paper. Actually, there are published papers on the evaluation of the Project P.A.T.H.S. based on weekly diaries [28, 29]. In this paper, the findings based on weekly diaries in eight datasets were aggregated in order to give a picture regarding the effectiveness of the Project P.A.T.H.S. in Hong Kong. In short, the purpose of this paper is to give a descriptive picture on the views and experiences of the program participants.
2. Methods
2.1. Participants and Procedures
From 2005 to 2009, a total of 244 secondary schools participated in the P.A.T.H.S. Project. Among all the participating schools, 12 schools that joined the secondary 1 level of the Project, 12 schools that joined the secondary 2 level, and 10 schools that joined the secondary 3 level were randomly selected and invited to take part in eight research studies. In each of the selected schools, the school contact person was asked to randomly invite some students (from the whole grade or in some classes) to write a piece of journal in the form of a weekly diary in order to reveal their perceptions and feelings after joining the Tier 1 Program of the project. The students were informed of the purpose and confidentiality of this study, and their consent for participation was sought prior to data collection. The title of the reflective journal was “Participating in the Tier 1 Program of the Project P.A.T.H.S.: Experiences and Feelings.” The reflection was expected to be not less than 200 words in Chinese, and it should be related to the students’ experiences, feelings, and comments in connection with their participation in the Tier 1 Program. The students could either complete it at home or during class time. The total number of students’ diaries received was 1,138, comprising 369 written by secondary 1 students, 486 written by secondary 2 students, and 283 written by secondary 3 students (see Table 1). As the cases were randomly selected and the response rate was 100%, the recruited sample could be regarded as respectable. No secondary 1 students were recruited in the 2007/2008 school year because several cohorts of data had been collected.
While the basic research design of the studies generating the datasets was primarily qualitative in nature (i.e., written comments on the program, instructors, and benefits), it is possible to use quantitative methods (such as counting and frequency analyses) to create a descriptive profile of the findings [28]. In the present study, the secondary data analysis approach was adopted by integrating and synthesizing the data generated from the different databases.
2.2. Data Analyses
In each of the eight studies, the data were analyzed using general qualitative analyses techniques involving three steps [30]. First, relevant raw codes were developed for words, phrases, and/or sentences that formed meaningful units at the raw responses level. Second, the codes were further combined to reflect higher-order attributes at the category of codes level (i.e., themes and subthemes). Third, the categories of codes were further analyzed to reveal the broader themes at the thematic level. Three main themes were identified from the weekly diaries: views on the program, views on the instructors, and perceived benefits of the program. Both intra- and interrater reliabilities on the coding regarding these views were calculated, and the results were satisfactory (85–100%). In the present analyses, because the researchers designed the program of the Project P.A.T.H.S., they were conscious of their own biases and expectations on the program to be effective. This paper reports the analyses based on the aggregated findings of these eight studies. The major categories and subcategories of responses are presented in the tables, and narrative illustrating the themes can be seen in the Section 3.
3. Results
Responses regarding the students’ perception of the program and the instructor, as well as the perceived benefits of the program, are presented in this paper. The integrated findings are presented in Tables 2–4. As shown in Table 2, there was a total of 2,290 meaningful units regarding students’ views on the Tier 1 Program implemented from 2005 to 2009. The responses were categorized into “overall impression,” “program content,” “learning process,” “program arrangement,” and “other comments.” About 81% of the total responses was coded as positive. Most of the respondents had positive views on the “overall impression,” followed by “learning process” and “program content”. There were 85.4, 77.2, and 83.1% of secondary 1, 2, and 3 students who reported positively on the program, respectively, although some negative comments were recorded. Three cases illustrating the respondents’ perceptions of the program at each grade are presented below.
Student A (Secondary 1, 2006/2007)
“This is my first time joining the P.A.T.H.S. Project. I learned about emotion, choice and other knowledge about development from the program … I personally think that this program has great impact on me and my classmates. We could learn via the worksheet activities as well as from the advice given by the classmates. What we learned are closely related to reality which could be applicable to daily life. Therefore, I think the program is very meaningful.”
Student B (Secondary 2, 2007/2008)
“After completion of the Tier 1 Program in the first semester, I felt I became more mature in interpersonal communication and personal growth. We studied various problems such as teenagers’ learning attitudes and hedonism during class. Through games or group discussion, we learned how to analyze and acquire correct information, and how to face difficulties and remain positive under adverse circumstances. I like this program very much because all the program content is closely related to common teenage problems, such as body shaping and substance abuse, which is closely related to our daily life. If we do not have good virtues and healthy mind, we will follow the wrong track easily. Therefore, the program serves as not only a light in the dark but also a teacher who teaches and guides us to the right way.”
Student C (Secondary 3, 2008/2009)
“Fortunately, I joined the Project P.A.T.H.S. from Secondary 1 to Secondary 3. This is one of my favorite subjects. The atmosphere during the class was relaxing that made the subject become more acceptable. Moreover, I really liked the design of the program. Teachers and social workers used multiple resources for teaching in order to guide the students to think from different angles and make judgment. Problems such as school orientation, conflicts with parents and peer relationship that troubled me a lot were discussed in the class. I could solve these problems by applying the skills I learned from the class. Obviously the project works.”
There were 558 responses regarding “views on instructors,” which were categorized into “overall impression,” “teaching performance,” “teaching attitude,” and “others” (see Table 3). About 90% was positive responses. Most of the respondents had positive views on instructors’ “teaching performance.” Several cases illustrating the respondents’ views on the instructors are shown below.
Student D (Secondary 1, 2006/2007)
“I remembered when I first joined the Project P.A.T.H.S., I felt very excited … Miss Wong was our instructor. Although the topics of each P.A.T.H.S. session were different, the goal was the same—to help students to develop good interpersonal relationship. The instructor would encourage the students to share their opinions on a topic and then asked other students to discuss or analyze each student’s viewpoints. The instructor would also arrange some activities such as group discussion and role-play. These activities promoted the relationship among classmates in only one year’s time.”
Student E (Secondary 3, 2008/2009)
“There was a period of time when I lost self-confidence due to a failure. I kept myself in the corner of my world and could not find a way out until I heard some sharing from the instructor. She said, “Life is like an obstacle race. No matter what obstacles you encounter during the race, once you pass the finishing line, you will have victory.” I suddenly realized that self-pitying did not help anything. Rather, I should regain my strength to face new challenges. The words from the instructor become my motto, which drives me to keep moving forward.”
The perceived benefits of the program to the students are shown in Table 4. The 2,986 meaningful units were categorized into six main categories (i.e., societal, school, familial, interpersonal, personal levels of competence, and others). Most of the responses were related to the effectiveness of the program on a personal level (,936), particularly in the subcategory of “positive self” (, e.g., “enhanced self-understanding”), followed by “general gains” (, e.g., “acquired skills/knowledge”) and “cherishing life” (, e.g., “promoted reflection of life”). In addition, many respondents reported that they acquired interpersonal competence (), which could be categorized into “general interpersonal competence” (e.g., “got along with others”) and “specific interpersonal competence” (e.g., “learned to accept, care, trust, respect, and understand others”). Some narratives based on the respondents revealed that the program was perceived positively by the program participants.
Student F (Secondary 1, 2006/2007)
“I learned much useful knowledge in this program. Firstly, I learned many interpersonal skills such as how to get along with classmates, teachers, friends, senior people and family members. I also learned to appreciate others and myself. I knew myself much more than before. By realizing my strengths and weaknesses, I was able to improve myself. In addition, I learned to distinguish between what I should do and what I should not. Most important of all, the program enabled me to meet new friends and improve my communication skills. All these skills will be very useful for my future development.”
Student G (Secondary 2, 2007/2008)
“When I looked back, I really learned a lot of things. I felt I truly grew up. In familial aspect, I understood that family members should mutually help each other and care for one another. I need to help doing the family chores when I have time. As parents have been taking care of us whole-heartedly, we have to repay and be thankful to them. I learned the importance of communication. Now I would spend more time to chat with my parents and share my feelings with them. In doing so, my parents were reassured and became less worried about me … In social aspect, I knew that we should follow social disciplines such as keeping the voice down in public area to avoid interrupting other people … In global aspect, I learned that there were many people in the world living in poverty. We should do whatever we could do to help these people.”
Student H (Secondary 3, 2008-2009)
“I could still remember when I first participated in the Project P.A.T.H.S., I did not understand why we should sacrifice two sessions of lesson for studying this program. However, I was gradually involved in it. This program provided me an opportunity to know more about my teachers and classmates … Although the three-year program is going to end, we still have a lot to learn throughout our lives. Among the units I have learned, the most impressive topic is “Life and Death”. “Life and Death” is actually not far away from us. Remember the Wenchuan earthquake last year? Life of the students in Sichuan was changed in only one second. As Hong Kong students, we should make deepest reflection—bliss is not taken for granted!”
4. Discussion
The purpose of this study was to evaluate the Tier 1 Program of the Project P.A.T.H.S. using findings based on weekly diaries collected in the experimental and full implementation phases (2005–2009) of the project. There are several characteristics of this study. First, a large sample of participants (,138 students) was involved. Second, eight datasets collected across three grades at different points in time were analyzed. Third, this is the first known scientific evaluation study of a positive youth development program utilizing student diaries based on different cohorts in China. Fourth, this is also the first evaluation study utilizing diaries based on such a large sample of participants in the global context. Finally, the data were analyzed in both quantitative and qualitative ways. As far as the design and data collection of the studies are concerned, the study has the characteristics of naturalistic inquiry, inductive analysis, holistic perspective, qualitative data, unique case orientation, and contextual sensitivity. Regarding data analyses, both qualitative (e.g., coding and categorizing) and quantitative (mainly counting) analyses were used.
Based on the weekly diaries written by the students, the studies reported in this paper showed that the participants generally had positive perceptions of the Tier 1 Program of the Project P.A.T.H.S. and the instructors. The findings presented in Tables 2 and 3 suggest that the participants had positive impressions of the program, and they appreciated the program content and learning process involved. In addition, Table 4 shows that the participants overwhelmingly regarded the program as beneficial to the cognitive, emotional, social, and productive quality of life of the program participants. For example, a sizable number of responses were related to positive self-image after joining the program (Table 4). In addition, many participants remarked that their interpersonal competencies were enhanced after joining the program. In short, the findings in Table 4 basically suggest that the program is beneficial to the development of the program participants.
The above findings are generally consistent with previous research findings in the experimental and full implementation phases, and they basically concur with both the quantitative and qualitative findings based on objective outcome, subjective outcome, qualitative, and process evaluation findings (e.g., [24–27, 31]). In short, the present findings provide additional evidence for supporting the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. in Hong Kong. From the perspective of triangulation, the existing evaluation findings suggest that the effectiveness of the Tier 1 Program of the Project P.A.T.H.S. is supported by evaluation data collected from different sources and by different strategies.
The present findings demonstrate the utility of using student diaries to evaluate positive youth development programs. By asking the program participants to reflect on their experiences, subjective perceptions and perceived benefits of the program can be properly understood in a less mechanical and artificial manner. Such an approach can help to capture the dynamic nature of the perceived qualities and effectiveness of the program. In addition to its holistic emphasis and ease of data collection, Maxwell [32] commented that “qualitative research is a rigorous means of investigating causality” (page 3). Slayton and Llosa [33] also remarked that “qualitative methods can confirm that it is actually the program that is responsible for the effect” (page 2544). Obviously, the use of reflective journals enjoys the strengths of qualitative studies in assessing the changes in quality of life among the participants after joining the program.
Shek et al. [34] proposed that several principles should be maintained in order to enhance the quality of qualitative research. As a qualitative study, the following attributes are intrinsic to this study. First, a general qualitative orientation was adopted. Second, recruitment processes for the participants and justifications for the number of participants are described. Third, details of the data collection are given. Fourth, issues of biases and ideological preoccupation are addressed. Fifth, inter- and intrarater reliabilities information is presented. Finally, the categorized data are kept in a systematic filing system in order to ensure that the findings are auditable.
Nevertheless, several limitations of the study should be noted. First, for students who are sensitive about the issue of “invasion of privacy” [12], the use of a weekly diary may be regarded as an obtrusive evaluation device. Second, as the diary method is written in nature, students who are not good at using words to express their experiences may write very little. This problem may be particularly acute in schools that admit students with poor academic performance or literacy. This is also a problem in the Chinese culture where Chinese people may lack the language to describe their feelings. Third, as there was only one occasion on which the participants expressed their views in a one-way manner, it was not possible to have dialogues with the informants to further understand some of their views. In particular, theoretical sampling approaches could be attempted in future studies.
Fourth, although the present findings are interpreted in terms of the positive program effects and experiences of the program participants, it should be noted that there are several alternative explanations. These alternative explanations should be considered because high proportions of responses were positive across the eight studies. The first alternative explanation is that the students were afraid of being punished by the program implementers if they did not respond in a favorable manner. However, this possibility can be partially dismissed because the teachers did not mark the weekly diaries and the identities of the students were basically anonymous. The second alternative explanation is that the students consciously acted in a “nice” manner to help the workers to illustrate a positive program effect. However, this alternative explanation can be partially dismissed because negative comments were recorded and students are normally encouraged to express their views honestly. The third alternative explanation is that the high proportion of positive responses observed, were in fact, biased responses. However, this alternative explanation can also be dismissed because the schools and students were randomly selected. The fourth alternative explanation is that because there is no control group, natural developmental maturation may account for the positive evaluation findings. That is, through maturation, the participants may see things in a more favorable manner. Furthermore, threats to internal validity in the study should be realized.
Finally, although the 11 principles proposed by Shek et al. [34] were upheld in this study, peer and member checking (principle 8) was not carried out because of time and manpower constraints. Despite these limitations, this study provides pioneering evaluation findings across samples and time that support the positive nature of the Project P.A.T.H.S. and its effectiveness in promoting holistic youth development among Chinese adolescents in Hong Kong.
Acknowledgments
The preparation for this paper and the Project P.A.T.H.S. were financially supported by The Hong Kong Jockey Club Charities Trust. The authorship is equally shared between the first and second authors.