Abstract

Research on the relationship between after-school program participation and student outcomes has been mixed, and beneficial effects have been small. More recent studies suggest that participation is best characterized as a multidimensional concept that includes enrollment, attendance, and engagement, which help explain differences in student outcomes. The present study uses data from a longitudinal study of after-school programs in elementary schools to examine staff ratings of student engagement in after-school activities and the association between engagement and school outcomes. The factor structure of the staff-rated measure of student engagement was examined by exploratory factor analysis. Multiple regression analyses found that student engagement in academic, youth development, and arts after-school program activities was significantly related to changes in teacher ratings of academic skills and social competence over the course of the school year and that students with the greatest increase in academic skills both were highly engaged in activities and attended the after-school program regularly. The results of this study provide additional evidence regarding the benefits of after-school programs and the importance of student engagement when assessing student outcomes.

1. Introduction

Recent reports indicate that over eight million K-12 children participate in after-school programs [1]. Research concerning the effects of after-school programs on student social and academic outcomes has been mixed. Several large-scale evaluations of federally funded 21st Century Community Learning Centers after-school programs found no effects on student academic achievement among elementary or middle school students [2]. Other studies have found more positive results though, with meta-analyses indicating small but significant effects of after-school program participation on reading and mathematics achievement [3] and on personal and social skills [2].

Given this mixed bag of findings and overall small effects, researchers have focused on identifying factors that may explain the ways in which participation in after-school programs affects student outcomes. These studies have examined program and individual student factors and suggest that a combination of individual, program, and contextual factors is important when assessing the impact of after-school programs [24].

According to the Harvard Family Research Project model, variability in the effects of after-school programs on academic and social outcomes is best explained by variability in student participation, a three-part construct that includes student enrollment, attendance, and engagement [5]. Engagement has affective, cognitive, and behavioral components and entails enjoyment of, interest in, and sustained attention and effort focused on an activity [57]. This multidimensional construct differs from previous conceptions of student participation that equated participation with enrollment or attendance but did not consider the combination of these three factors (enrollment, attendance, and engagement) when assessing student outcomes [5]. Subsequent research has confirmed the importance of considering attendance and engagement, in addition to enrollment, when assessing outcomes related to after-school programs [2, 8].

Despite the theoretical importance of student engagement in after-school programs as a key facet of program participation, only a few empirical studies have incorporated engagement as a predictor of outcomes associated with after-school participation. There is however a larger body of literature on the effects of student engagement during the school day. This literature—which conceptualizes engagement as multidimensional and having cognitive, emotional, and behavior components—shows that engagement in school is robustly associated with higher academic achievement and lower risk of school dropout (e.g., [912]). Theoretical models of student engagement also highlight the potential benefits of being highly engaged in school for social relationships and emotion regulation [9], although such social competence outcomes have not been as widely documented as academic ones, notably achievement and school completion.

Studies that have examined engagement in after-school activities find that it is also associated with academic and social benefits. Mahoney et al. [7] found that student engagement, as rated by program staff, interacted with after-school attendance in predicting academic outcomes among elementary school children: student engagement increased the positive association between after-school participation and classroom performance. Another study of middle school students found that student engagement as rated through experiential sampling methods partially explained the relationship between program participation and social competence [8]. These studies suggest that engagement may be associated with student outcomes in multiple ways (i.e., additive and interactive effects) and point to the need for additional empirical examination of the role played by student engagement in after-school programs to understand how program participation can make an impact on student outcomes [5, 8].

In the present study, we use data from a longitudinal study of after-school programs in elementary schools to examine staff ratings of students’ engagement in after-school program activities and how engagement is associated with academic and social outcomes in school. Most empirical investigations of student engagement during school or in after-school activities have relied on either student or teacher/staff reports, with similar patterns of findings across reporters [79, 11]. In the present study, the rationale for our reliance on staff reports was threefold: (a) to ensure that the assessment of engagement focused on after-school activities (i.e., staff did not observe students during the school day), (b) because the programs in the study served a wide range of grades, as young as prekindergarten, and (c) to have an assessment easily administered by programs as part of self-assessments.

Using data from the first year of the study, we examine the factor structure program staff-rated engagement measure and how its factors correlate with elementary school teachers’ ratings of classroom behavior. Using data from the second year, we test the unique effects of student engagement in after-school activities on change in academic skills and social competence over the course of the school year, as well as the interaction between engagement and another facet of program participation: program attendance. Based on Weiss et al.’s [5] framework for after-school program participation and prior research [7, 13], we hypothesize that student engagement in after-school activities is associated with increased academic skills and social competence, particularly when coupled with regular program attendance.

2. Method

2.1. Program and Participants

All participants were enrolled in one of nine after-school program sites located in elementary schools in the inner-circle suburbs of a Midwestern city. Over the course of the two years of the study, most sites were in their first two full years of operation. Programs are designed to follow a community school model, which sees the school as the center point of the community where youth, families, and community residents work as equal partners with schools and other community institutions to develop programs and services. Community schools aim to foster strong partnerships between schools, families, and community partners, share accountability for results, set high expectations for all, build on the community’s strengths, and embrace diversity. Programs provided out-of-school care with an emphasis on activities designed to foster student achievement and social emotional growth. The nine program sites were housed in schools from three public school districts and served children from prekindergarten through sixth grade. At each site students participated in a range of after-school activities including academically focused activities and youth development activities such as Girl Scouts, athletic activities, arts activities, homework help, and free play. After-school time typically began with snack time and homework help, was followed by students splitting up into various structured activities, and ended with free play. During the 2009-2010 school year, sites varied in terms of the percentage of students receiving free or reduced lunch ranging from 67.7% to 90.3% (), staff size (4–8 staff members), and program quality (see below). During the 2010-2011 academic year, percentage of students receiving free or reduced lunch ranged from 66.2% to 90.5% (M = 81.2%). On average across the three districts student demographics were as follows: 45.9% Black, 40.2% White, 11.8% Hispanic, 1.6% Asian, and 0.6% Indian. On average, staff members were 33 years old (range = 18–80 years old) and had 2.49 years of experience as after-school staff (range = 1 month to 12 years). Staff members were primarily female (72%) and their level of education ranged from less than high school (1.9%) to Bachelor’s degree (15.1%), with the majority (54.7%) receiving a high school diploma or GED, followed by an Associate’s degree or technical school/certification (28.3%).

2.2. Procedure

Survey data were collected from after-school program staff and elementary school teachers as part of a larger study. These data were collected by the program and entered into database used for ongoing program assessment and monitoring. Deidentified data files were shared with researchers by the program. Data from surveys administered in April 2010 were used to test the factor structure of the engagement measure. Data collected in October and April from the following school year (2010-2011) were used to examine the effects of after-school program engagement on changing classroom social competence and academic skills. Survey administration roughly matched onto the end of the first and third quarter marking periods. During the first year of the study (spring 2010), program staff completed survey ratings of engagement for 667 students. Of these students, 530 also had survey ratings from their elementary school teachers. During the second year of the study (2010-2011), data from both program staff and elementary school teacher surveys were available for 384 students. See Table 1 for breakdown by ethnicity, grade, and gender.

Program quality was assessed through on-site observations conducted by graduate students and research assistants, who underwent training using videos and live observations in the observational instrument used in this study. Program activities were observed in teams of two and data were collected over two afternoons at each site. Researchers arrived on-site shortly before the beginning of the after-school program. After introductions to staff and discussion about the intended curriculum for the day, the observers positioned themselves as unobtrusively as possible to coobserve one activity at a time. At the beginning of an activity, each observer recorded details about the setting and content of the activity, as well as the grade levels, number of staff, and number of participants. Evaluators observed activities for 15 minutes and after each activity, the observers compared their data to monitor interrater reliability. Activity observations were collected in the spring of each year.

3. Measures

3.1. Student Engagement

After-school program staff rated student engagement during a range of after-school activities using the Staff-Rated Student Engagement Survey. This measure was adapted from the School Engagement Measure by Fredericks et al. [11, 14] to fit the after-school context. The School Engagement Measure asks youth to self-report on three areas of engagement: behavioral, emotional, and cognitive [14]. These areas address the different aspects of engagement looking at participation and attentiveness, the appeal of and positive and negative reactions to the information taught, and the personal investment and effort towards advancing one’s own learning.

For the current study, the wording of the items was changed to more accurately describe after-school activities and from first person to third person to allow for ratings by program staff. Furthermore, cognitive engagement was excluded because it originally asks students about what they do at home and staff members would not know the extent that students follow-up at home on topics introduced in the program activities. Program staff members indicated how often (never = 1, on occasion = 2, some of the time = 3, most of the time = 4, and all of the time = 5) each student exhibited the following—pays attention, seems interested in the subject, is on task, seems to have fun, gets in trouble, and seems bored—in four types of activities, as classified by the program (academic, youth development, athletic/sports/physical, and arts and crafts). Higher scores indicate a student was more engaged in activities during the after-school program. More detail on the scoring of the scale is presented in the Results.

3.2. Program Quality

Observational measures of program quality were assessed by the Out-of-School Time Observational Instrument (OST), which rates program quality by the extent to which activities are sequenced, active, interpersonally focused, and explicit (SAFE; see [15, 16]). This tool was chosen because it reliably measures relational aspects of the before- and after-school programs, which are factors that have demonstrable effects on positive program outcomes. Its observation scales follow a framework that is closely aligned with the SAFE features that are present in high-quality after-school programs [13]. The measure targets the quality of youth-staff relationships as well as the level of engagement of youth in the program activities. Its methodology is ideal because it allows researchers to unobtrusively observe program activities while generating a rich array of data. After a 15-minute observation period per activity, the observers rate the activity in four key domains: Youth Relationship Building, Youth Participation, Staff Relationship Building, and Staff Instructional Strategies. The items are rated on a 7-point scale from 1 = exemplar is not evident to 7 = exemplar is highly evident and consistent, with a 5 = exemplar is evident or implicit. Items can be grouped together to form the sequenced, active, interpersonally focused, and explicit scales, which are then averaged together for an overall program quality score (SAFE score). The report of validation for this measure has shown high reliability and internal consistency, construct, and predictive validity for these scales as well as high concurrent validity in relation to survey data [1521]. The measure also demonstrated good internal consistency in the current study for both spring 2010 (Cronbach’s α = 0.86 for overall SAFE rating) and spring 2011 (Cronbach’s α = 0.78 for overall SAFE rating). A total of 44 activities were observed in spring 2010 and 54 activities were observed in spring 2011 across the nine schools in three districts. As indicated in Table 2, most program sites were of moderate quality, with OST overall SAFE scores ranging from 2.71 to 4.50.

3.3. Social Competence

Social competence was measured by the Fast Track Social Competence Scale—Teacher Version  [22]. The survey was distributed to participating students’ elementary school teachers. Teachers rated student’s behavior regulation (e.g., “Gets angry when provoked by other children”) and empathy (e.g., “Provides help, shares materials, and acts cooperatively with others”) on 12 items. These 12 items were combined to form an overall social competence score. The rating system for these items was on a 5-point scale ranging from 1 = not at all true to 5 = very much true. The measure has shown good internal consistency in past studies [23] and had good consistency in the current sample (Cronbach’s α = 0.96 for spring 2010, Cronbach’s α = 0.95 for fall 2010, and Cronbach’s α = 0.93 for spring 2011). Each student’s social competence score reflects average teacher rating across the items at each measurement time (spring 2010, fall 2010, and spring 2011). Higher scores indicate greater social competence in the classroom.

3.4. Academic Skills

Academic skills were measured by the Fast Track Social Competence Scale—Teacher Version (CPPRG, 1995). Elementary school teachers rated students’ academic proficiency (e.g., “Is able to read grade level material and answer questions about what they have read”) on 5 items. The rating system for these items was on a 5-point scale ranging from 1 = not at all true to 5 = very much true. The measure has shown good internal consistency in past studies [23] and had good consistency in the current sample (Cronbach’s α = 0.91 for spring 2010, Cronbach’s α = 0.93 for fall 2010, and Cronbach’s α = 0.92 for spring 2011). Each student’s academic skills score reflects average teacher rating across the items at each measurement time (fall 2009, spring 2010, fall 2010, and spring 2011). Higher scores indicate greater academic skill in the classroom. Teacher ratings of academic skills were chosen over academic grades as an outcome to provide a standardized assessment of student achievement, each of the three districts used given a different grading metric, and even within district, different metrics were used for different grade levels.

3.5. After-School Program Attendance

After-school program attendance was recorded by program staff members and reported to program directors. After-school attendance reflects the number of days each student attended the after-school program during the respective school year. Average attendance for the first year of the study was 147.3 days (SD = 36.8) and 148.5 days for the second year of the study (SD = 34.4).

3.6. Covariates

In addition to the dummy codes created to control for differences between sites, all analyses controlled for gender (female = 1, male = 0), grade level (kindergarten—6th grade), and ethnicity (African American = 1, other = 0).

4. Results

Preliminary analyses examined the factor structure of the Staff-Rated Student Engagement Survey and how its factors correlated with teacher ratings of students’ academic skills and social competence. Next, hierarchical regression was used to examine whether student engagement in after-school programs was positively associated with change in academic skills and social competence from fall to spring of the school year.

4.1. Factor Structure of the Student Engagement Measure

An exploratory factor analysis was performed on staff ratings of student engagement from spring 2010 to identify latent constructs within the 24-item Staff-Rated Student Engagement Survey. Common factor analysis was performed on the 24 items in the Student Engagement Survey. A promax rotation, which allows for a relationship between factors, was utilized due to the possibility of overlap between potential factors.

Parallel analysis was used to select the number of factors [24]. In the parallel analysis, eigenvalues generated from the data were compared to eigenvalues from 500 randomly generated datasets of the same sample size. According to the results, three factors had eigenvalues greater than the average and 95% percentile of randomly generated eigenvalues (see Table 3), indicating a three-factor solution. These three factors accounted for over 65% of the variance in the engagement items (Factor 1 = 39.86%, Factor 2 = 16.88%, and Factor 3 = 8.46%).

The pattern matrix of factor loadings is presented in Table 4. Twelve items related to engagement in academic, youth development, and arts activities loaded on the first factor. Four items related to engagement in athletic activities loaded onto a separate factor (Factor 3). The eight items that assessed students’ tendency to get in trouble or get bored across all activities loaded on a separate factor (Factor 2); we labelled them as indicators of disengagement (Factor 2). Factors were intercorrelated. Factor 1 and Factor 2 correlated with , Factor 1 and Factor 3 correlated with , and Factor 2 and Factor 3 correlated with .

The mean of staff ratings across the eight items in Factor 1 was calculated for each student to represent engagement in academic, youth development, and art activities (M = 3.71, SD = 0.66). The scale had good internal consistency, Cronbach’s α = 0.95. The mean of ratings across the four items in Factor 3 was calculated for each student to represent engagement in athletic activities (M = 3.84, SD = 0.71). This scale also had good internal consistency, Cronbach’s α = 0.91. The mean of ratings across the eight items in Factor 2 was calculated to represent disengagement (M = 2.28, SD = 0.91). This scale also had good internal consistency, Cronbach’s α = 0.91.

Engagement in academic, youth development, and art activities was correlated with teacher-rated academic skills, , , and social competence, , , in the spring of 2010. Engagement in athletic activities was more modestly correlated with teacher-rated academic skills, , , and social competence, , , in the spring of 2010. Disengagement was negatively correlated with teacher-rated academic skills, , , and social competence, , . Program attendance was not correlated with disengagement, , , with engagement in athletic activities, , , or with engagement in academic, youth development and arts activities, , .

4.2. Effects of Engagement on Change in Academic Skill and Social Competence

Following Mahoney et al. [7], for data from the 2010-2011 school year, the mean of staff ratings across the engagement items and the two assessment periods (fall and spring) was calculated for each student to represent (1) their mean engagement in academic, youth development, and art activities across the school year and (2) their mean engagement in athletic activities across the school year. Fall and spring ratings were combined to represent average engagement in after-school activities across the school year. Items loading on Factor 2 were removed from the survey in the 2010-2011 school year, so analyses from this year did not include disengagement.

Table 5 presents correlations among the study variables. As indicated in the table, younger students and girls were rated being more engaged in academic, youth development, and art activities. There were also significant differences in the levels of staff-reported engagement across sites for engagement in both academic, youth development, and art activities, (7, 378) = 15.45, , and athletic activities, (7, 378) = 9.34, . Differences across sites were likely due to several factors including variability in program quality, demographic differences in the students enrolled, and biases in how program staff completed the engagement ratings from site to site. Student demographic characteristics and site were included as covariates in the subsequent analyses to control statistically for their potentially confounding effects.

Two sets of multiple linear regressions were computed to assess whether student engagement in after-school programs was associated with social and academic classroom outcomes over the course of the 2010-2011 school year. Regressions assessed the relationship between the two domains of staff-rated student engagement identified during the previous EFA analysis—engagement in academic, youth development, and art activities and engagement in athletic activities—and the two outcomes of interest: teacher-rated academic skills and social competence. Regressions followed the same order of variable entry and included the same covariates. Spring scores on each outcome were regressed on fall ratings of social competence and academic skills, additional covariates (ethnicity, sex, grade, and dummy codes representing program sites), program attendance, the staff-rated engagement variables, and product terms representing the interaction between each engagement variable and program attendance. By controlling for fall levels of social competence and academic skills, the spring outcomes are interpreted as residualized change scores, which assess rank order change in academic skills and social competence relative to other students in the sample [25]. Results for both sets of regressions are presented in Table 6.

4.2.1. Teacher-Rated Academic Skills

When the student engagement variables were entered simultaneously with ethnicity, grade, sex, program site, attendance, and fall 2010 ratings of academic skills and social competence, there was a statistically significant effect of student engagement in academic, youth development and art after-school activities on increased academic skills during the 2010-2011 school year. Although the effect of engagement in academic, youth development and art after-school activities was statistically significant, it was small, uniquely explaining 1% of the variance in academic skills. There was not a statistically significant effect of engagement in athletic activities on academic skills.

There was also a significant interaction of engagement in academic, youth development, and art activities with program attendance, which uniquely explained less than 1% of the variance in academic skills. The interaction is graphed in Figure 1. At average levels of program attendance, there was a small effect of engagement on increased academic skills, β = 0.14, , and the more frequently students attended the program, the stronger the effect of engagement on academic skills will be. For students who attended the program almost every day during the school year (1 SD above the mean), the effect of engagement on academic skills was β = 0.23, . Conversely, for students who only attended the program infrequently (1 SD below the mean), there was not a statistically significant effect of engagement in academic, youth development, and art activities on academic skills, β = 0.10, .

4.2.2. Teacher-Rated Social Competence

When entered simultaneously with ethnicity, grade, sex, program site, attendance, and fall ratings of academic skills and social competence, student engagement in academic, youth development, and art after-school activities was significantly associated with increased social competence in school during the 2010-2011 school year. This effect uniquely predicted 2% of the variance in social competence. There was not a statistically significant effect of engagement in athletic activities on social competence nor were interactions between engagement and program attendance detected.

5. Discussion

The results of this study provide additional evidence regarding the importance of student engagement in after-school program activities. The current study corroborates previous research [7, 8] which suggests that student engagement in after-school activities is related to increased academic and social skills among elementary school students. These findings, as well as previous ones (e.g., [7]), speak of the value of conceptualizing participation in after-school programs in ways that go beyond program enrollment to encompass the interplay of program attendance and engagement in program activities [5]. More importantly, the current study provides evidence that the students who benefit the most academically from after-school programs both attend regularly and are highly engaged in specific types of activities (academic, youth development, and art).

Larson [6] argued that after-school activities are critical for supporting students’ engagement because they tend to be both structured and enjoyable. While engaged in high-quality after-school activities, students may experience higher degrees of intrinsic motivation, place more effort into their activities, and feel less bored [26]. Students are thus able to reinforce skills such as self-regulation, sustained attention, collaboration, and goal-directed behavior [6], skills that may translate into the classroom. It is also possible that engagement in school-based after-school programs may translate more directly to improved performance and behavior in the classroom during the school day, because even though the activities are different during after-school, the general setting is the same as during the school day. Future research should examine how engagement in after-school programs offered in different settings translates to achievement and behavior in school.

Another finding of note is that engagement in academic, youth development, and arts activities clustered together and explained a majority of the variance in student engagement, separately from engagement in athletic and physical activities. These engagement scores were validated by their relationship with teacher ratings of academic skills and social competence. Past research has tended to measure student engagement across activities but these findings suggest that engagement in certain types of activities is more related to academic skills and social competence. In particular, academic, youth development, and art activities may be more likely to provide a structured and challenging environment that actively involves children, characteristics that have been associated with program quality [13].

Alternatively, engagement in athletic activities may play an important role in the effects of after-school participation on nonacademic outcomes, such as obesity prevention [7]. Indeed, meta-analytic findings indicated that participation in after-school programming can have a positive impact on physical activity level and other markers of physical health [27]. Engagement in athletic activities may help explain the effects of after-school program participation on these types of health outcomes.

Results from this study should be interpreted with some caution. It is important to note that the effects reported were relatively small. In addition, because it is not possible to experimentally manipulate students’ level of engagement, the direction of effects linking engagement in program activities with school outcomes cannot be isolated. Given the extent to which staff ratings of student engagement in after-school activities were linked to students’ behavior and performance in the school day, continued focus on engagement is key to understanding how program participation can lead to beneficial student outcomes.

Another weakness concerns the use of staff-report to measure student engagement in after-school activities. Staff ratings reflect their perceptions of student affect, behavior, and motivation but it is difficult to equate these perceptions or observations with students’ psychological states. Lastly, this study did not include enough program sites to be able to systematically address the role of program quality, which has been linked to student engagement in past studies [4]. A substantially larger number of programs than in the current study would be necessary to systematically assess how facets of observed program quality are associated with variability in student engagement across program sites [28]. Past research on program quality suggests that program activities that are structured, active, interpersonally focused, and explicit are related to increased student engagement and positive social and academic outcomes [13]. These features may provide direction for after-school staff and help guide curricula designed to engage students in program activities.

In conclusion, this study provides additional evidence of the utility of a staff-rated measure focused on student engagement during academic, youth development, and arts activities. Student engagement in after-school programs has emerged as a key predictor of student outcomes, and the present study provides additional validation that a fairly brief staff-rated measure that is appropriate for assessing this construct among elementary school children can be a useful resource for evaluating the effects of after-school participation on academic outcomes. More importantly, this study suggests that individual student engagement in specific types of after-school activities is associated with positive student outcomes during the school day, which should provide additional evidence of the potential benefits of after-school programming and provide guidance regarding the types of activities that these programs should emphasize. Future studies should further examine the relationship between changes in after-school programming and subsequent changes in student engagement. Continued research should focus on identifying specific program, student and staff characteristics that are associated with improvements in student engagement in order to provide more concrete guidelines for after-school programs on ways to promote sustained and engaged student participation [29, 30].

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

This research was supported with funding from the Kansas City Local Investment Commission.