Abstract

This study analyzes the psychometric properties of the engagement to academic tasks (Comp-TA) questionnaire. Rigorous criteria were considered in its design and validation, such as theoretical review, expert judgment, pilot test, and exploratory and confirmatory factor analysis. The instrument was administered face-to-face and in remote modality to a convenience sample of 563 high-school students from schools in Concepción, Chile. The analyses showed a latent structure with a good fit to the data, consisting of 15 items that underlie the factors of academic engagement: behavioral, cognitive, and affective. The reliability analysis yielded an internal consistency of .92 according to the ordinal alpha. Guidelines for its use to deepen the educational process of high-school students are argued.

1. Introduction

The educational process is faced with constant challenges linked to the updating, revision, and development of pedagogical praxis. The latter requires responding to new requirements and characteristics of a population that has been transformed [1]. Political, economic, social, cultural, and technological changes have intercepted the people and their way of understanding the world, and this has generated that the reception of information by students is modified [2]. Teixes [3] mentions that some fundamental features of this new population of students are that they are permanently connected and expect rewards immediately and constantly.

Given the above, there is evidence of a disconnection between the characteristics of students and teaching practices, a disarticulation that Meller focuses on the lack of training students in 21st-century skills noting that in “Latin America, 19th-century schools coexist, with 20th-century teachers and 21st-century students” (2016, p.30). An example of this is stated by Cornellá and Estebanell [4] who observed that teachers maintain a traditional praxis. Vezub [5] adds that novice teachers tend to reiterate the strategies they visualize from their mentors. This discordance is compounded by the stagnant learning results during the last decade. In Chile, as in many Latin American countries, standardized measurement tests are applied [6]. The results obtained using this type of control have been qualified as insufficient in language and mathematics [7]. Faced with this reality, the authors agree on the need to update the teaching methods [8, 9].

But teachers’ praxis is not the only factor that affects academic performance. Other variables impact the result that the student may obtain: socioeconomic-cultural level, parents’ expectations of their children, parents’ level of formal education, gender, and work occupation [10]. However, many of them cannot be controlled or affected by educational actions. In this context, interest arises in investigating a construct that has the potential to address persistent problems in the educational setting such as low academic achievement, high dropout rates, and school boredom [11]. Lara et al. [12] mention that high academic engagement would lead to successful educational trajectories in the school system. Bae and Han [13] add that, in educational systems, there is a question of how to improve the quality and educational standards in schools and universities, emphasizing that it is necessary to know and understand how students spend their time and energy during their studies. The construct that has these features corresponds to academic engagement, which has the quality of being malleable, responds to contextual characteristics, and is subject to environmental change [14]. Therefore, it is a concept that allows predicting the learning outcomes obtained by the student and, indirectly, evaluating the practices performed by the teacher within the classroom, since as mentioned by Shernoff et al. [15], teachers have some control in the students’ engagement; this control lies in the configuration of learning environments for them. In addition, Boekaerts [16] affirms that engagement tends to be higher in classrooms where teachers deliver challenging and authentic tasks with opportunity for choice.

Academic engagement is a construct that was initially studied in work contexts and referred to the energy and passion that workers had towards their activity [17]. In education, it has become a prominent theoretical orientation and receives great attention from educators and researchers [18]. This concept is understood by Gutiérrez et al. [19] as the involvement of students to achieve academic accomplishments. It is also presented as the way students interact with their academic activities [20, 21]. Peña et al. [22] add that engagement corresponds to the physical and psychological resources dedicated to the school experience. Similarly, Fredricks et al. [11] understand it as a meta-construct that allows observing behaviors, cognition, and emotionality.

The academic engagement has been studied from various perspectives. According to Alrashidi et al. [18], there are two main theoretical orientations or approaches in the literature. The first approach argues that academic engagement is made up of three dimensions: (a) cognitive engagement, (b) behavioral engagement, and (c) emotional-affective engagement. The second approach argues that engagement is composed of (a) vigor, (b) dedication, and (c) absorption. Faced with this theoretical ambivalence, Christenson and Reschly [23] and Fredricks et al. [11] agree on the difficulty of understanding this construct, since being too broad and with considerable variability of definitions, there is a risk of trying to explain almost everything related to the educational experience of learners and, in the end, explaining nothing.

Besides, the diversity of definitions also brings difficulties in measurement [24]. Internationally, there are instruments aimed at students in primary education [25, 26], secondary education [27, 28], and tertiary education [22, 29], where each of them addresses different dimensions of academic engagement. For example, the Student School Engagement Scale with four dimensions (SES-4D) developed by Tomás et al. [30] measures four components of engagement: cognitive, behavioral, emotional, and agentic. This last component is defined as the student’s ability to set their own goals, allowing their active involvement [30]. Because of its recent origin, and although it has contributed to the theory of engagement, a greater number of antecedents addressing it are lacking [18].

This situation is not foreign at the national level. In Chile, Zapata et al. [31] designed and validated an instrument for tertiary education students and their conceptualization of engagement was associated with indicators such as learning strategies, quality of interactions, institutional support, and collaborative learning, among others. Parra and Pérez [32] developed an instrument aimed at university psychology students. Their theoretical model characterized engagement in three dimensions: vigor, dedication, and absorption. However, their findings were not empirically consistent with the proposed model, given that a bifactorial structure was obtained. Aspeé et al. [33] conducted their research work with higher education students. Their theoretical structure was formed by three dimensions: engagement-oriented to academic development, engagement-oriented to personal-integral development, and engagement-oriented to citizen development. On the other hand, the proposal of Lara et al. [12] consisted of defining engagement considering as a reference the level of engagement to studies, based on three dimensions: cognitive, behavioral, and affective. This theoretical framework was validated in a sample composed of primary and secondary education students.

The instruments explained above measure, for the most part, generic aspects of engagement. However, Fredricks et al. [11] point out that researchers must be clear about how they define engagement and at what level it is being measured. For this reason, it was decided to study this construct from a narrower perspective: from the activities and tasks addressed in the classroom.

Engagement linked to classroom tasks and activities is defined as a set of favorable student behaviors such as effort, enthusiasm, and initiative [34]. In this study, engagement is composed of three dimensions: (a) affective, (b) cognitive, and (c) behavioral. The affective dimension corresponds to affective and emotional bonds, and is understood as how the student faces school activities [14]. Shernoff et al. [15] add that this dimension refers to the emotions and affections of students facing their tasks in the classroom. The cognitive dimension is understood as the student’s investment and effort in their studies. Finally, the behavioral dimension is understood as the consistency of effort, attendance, homework, and desired academic behaviors [14, 15].

The importance of this research lies in the need of having a valid and reliable instrument, particularly one designed to be administered to secondary school students, which will allow studying, understanding, and deepening academic engagement focused on academic tasks. It becomes even more relevant considering that in the Chilean context, there are no instruments to measure commitment from this more limited perspective, evidencing a gap in the literature. Moreover, frequently education professionals and researchers denounce the lack of instruments to assess certain characteristics of their students, including psychosocial and behavioral aspects noted in classes [30]. As a consequence, this study seeks to provide the scientific and educational community with an instrument that has the particularity of measuring the engagement to academic tasks of high-school students.

Therefore, the objectives proposed in this study are two: (1) to design an instrument measuring the engagement to academic tasks of high-school students and (2) to validate the Comp-TA to analyze the psychometric properties of a proposed questionnaire for measuring the engagement to academic tasks of high-school students.

2. Method

2.1. Participants

The sample consisted of 563 students from 9th to 12th grades who voluntarily expressed their consent to participate. Concerning gender, the sample consisted of 245 boys (43.5%) and 318 girls (56.5%). The students belong to schools located in the district of Concepción, Chile.

Regarding the sample size, the recommendation of Ferrando and Anguiano-Carrasco [35] was considered, which suggests a size above 200 cases to evaluate the psychometric quality of an instrument. Therefore, a subsample of 329 participants was used to explore the underlying factor structure using exploratory factor analysis (EFA). To verify this structure, a confirmatory factor analysis (CFA) was used with a subsample of 234 participants.

2.2. Procedures and Ethical Considerations

The Comp-TA questionnaire was applied in person using pencil and paper, and remotely through a Google Form. In the face-to-face modality, educational institutions were contacted and informed about the purposes of the research. The evaluators applied the questionnaire emphasizing the importance of answering the entire instrument. At the same time, the confidential and anonymous nature of the treatment of the information provided was guaranteed. In the remote modality, a link to the questionnaire was sent to the students’ e-mails, who had the option of accepting to participate, thus expressing their assent to answer it. The form indicates and explains that the data obtained will be kept anonymous and only the answers will be used to carry out the corresponding statistical analysis by the researchers.

2.3. Design of the Instrument

The Comp-TA questionnaire was designed based on three instruments such as (1) Engagement towards homework in classrooms scale for students at the primary level of education [26], (2) School Engagement Scale [12], and (3) Academic Involvement Scale [22].

The main strengths of these instruments and the reason for their selection are as follows: (a) theoretical convergence, since the conceptualization and operationalization of the engagement construct are consistent with the approach and purposes of Comp-TA; (b) rigorous proposals in terms of both design and validation, which include factor analyses; (c) use of ordinal alpha, which is appropriate given the polytomous nature of the items, a quality present in the instrument of Lara et al. [12]; (d) adequate sample size since they used a sample of more than 200 participants to perform the factor analyses, a size that is in line with the recommendation followed in this study; and (e) adequate procedures, which enhance the rigor of the instruments. This last feature is observable, for example, in the use of back-translation as a tool to check the equivalence and cross-cultural adaptation in the translation of instruments [26], and use of focus groups to determine the understanding of items by students [12] and the relevance given to the literature review to determine the selection of basic foundations for the construction of scales.

Thus, the initial version of the Comp-TA consisted of 27 items that address three theoretically defined factors: (a) affective engagement, (b) cognitive engagement, and (c) behavioral engagement. Table 1 shows the origin of the items that constitute the initial version of the questionnaire. Likewise, its design considered a 5-level Likert scale, where 1 corresponds to Strongly Disagree and 5 to Strongly Agree.

2.4. Validation of the Instrument

The validation method considered three processes: (i) validation by expert judgment, (ii) pilot testing, and (iii) psychometric validation, which included an exploratory factor analysis and a confirmatory factor analysis.

2.4.1. Validation by Expert Judgment

Nine judges participated, who evaluated the items in the initial version of the Comp-TA based on four criteria: relevance, sufficiency, clarity, and coherence. Of the participants, six are Chilean and three are from Argentina, the United States, and Spain, with postgraduate studies, and background in education, psychology, sociology, and research methodology.

Among the contributions, first, the expert judges agreed in observing that two items were too similar, item 12 (I concentrate a lot when the teacher introduces new topics or tasks) and item 15 (I concentrate on what the teacher is explaining in class), so item 15 was eliminated.

Second, four items were modified due to their complexity and extension according to the judges’ perspective. For example, item 22 (In learning tasks, I always try to ask myself questions because they help me to understand central aspects of the topic) was changed to I ask myself questions to understand central aspects of the topic.

Third, six new items were added to improve the internal coherence of the scale: 3 items in the behavioral dimension that refers to peers or classmates, given that the instrument aims at measuring students’ engagement in the activities carried out within the classroom: I ask for help from my classmates when I do not understand, I keep silent to help in the concentration of my classmates, and I pay attention to the arguments given by my classmates; 2 items in the affective dimension related to the feeling of achievement upon completion of a task: I achieved new knowledge and I am able to complete them; and 1 item as a complement to the concretion of an objective, belonging to the cognitive dimension: I think about achieving the objectives I set myself at the beginning of the activity. These items respond to the recognition of absences and gaps found by the expert judges.

In this context, the second version of the Comp-TA consisted of 32 items. The judges proposed the addition of a general slogan to guide the students’ answers. In this way, participants indicate their degree of agreement or disagreement according to the phrase “When performing a task in class:” was included. For the response categories, they advised using a 7-level Likert scale, given the new guidelines for the construction of measurement instruments in which 1, means totally disagree; 2, somewhat disagree; 3, disagree; 4, neither agree nor disagree; 5, agree; 6, fairly agree; and 7, totally agree.

2.4.2. Pilot Test

As a result of the validation by expert judgment, the new version of the Comp-TA was administered to 45 high-school students. The pilot test made it possible to corroborate that the students understood the questions and also to determine the reasonable duration of the application, which did not exceed 16 minutes.

Both stages, expert judgment, and pilot testing provided evidence of content and process validity, leading to the final administration of the Comp-TA.

2.4.3. Psychometric Validation

To carry out the EFA, first, a comparison of the factorial solutions was made using the polychoric correlation matrix and Pearson’s correlation matrix. After obtaining similar results, it was decided to factor the polychoric matrix given the ordinal polytomous nature of the items. Then, as preliminary analyses, a normal distribution of data was evidenced, given that the data fluctuated between ±2 in skewness and kurtosis [36]. Next, the homogeneity index was studied.

Second, the degree of adequacy of the data for factor analysis was evaluated using Kaiser, Meyer, and Olkin (KMO) indicators and Bartlett’s test of sphericity. For factor estimation, the unweighted least squares (ULS) method was used. Regarding the number of factors to be retained, Horn’s parallel analysis and theoretical convergence were used. For the factor rotation procedures, oblique rotation, Promin, was chosen. As for the criterion for interpreting item saturation, the recommendation of MacCallum et al. [37] was followed, above 0.50 given the number of cases.

Third, to corroborate the underlying structure, a CFA was performed. The ULS estimator was used since it does not require assumptions about the distribution of variables. Next, global goodness-of-fit measures were used: chi-square , root mean squared error of approximation (RMSEA), nonstandardized adjustment index (TLI), and comparative fit index (CFI).

Finally, the reliability analysis was performed using the ordinal alpha. This estimator uses the polychoric matrix instead of the covariance matrix. Its use is recommended over Cronbach’s alpha because the latter should be applied for variables of a continuous nature [38].

Regarding the software used for data analysis, preliminary analyses were performed using Jamovi 1.2.16 software. Factor analyses were performed using FACTOR software version 10.9.02 for the AFE and SPSS AMOS version 23 for the AFC.

3. Results

3.1. Homogeneity Index

Using the first subsample (n= 329), the homogeneity analysis of the 32 items was performed. First, they were grouped by expected factors and then as a whole. Given that, in both conditions, the items met the criterion of obtaining a coefficient greater than or equal to .30, there was no elimination of items [39].

3.2. Exploratory Factor Analysis

The polychoric correlation matrix was used given the ordinal nature of the data [35]. The determinant of the matrix yielded a value of zero. This indicates that the degree of intercorrelation of the variables is very high [40]. The result of Bartlett’s test of sphericity indicated that the correlation matrix does not come from an identity matrix, . Likewise, the KMO index = 0.86 evinced that the correlation matrix is adequate for factor analysis.

The unweighted least squares (ULS) method was used since it does not require assumptions of normality. Regarding the number of factors to be retained, Horn’s parallel was used, which resulted in the suggestion of two factors. However, it was decided to use the number of factors according to the theoretical convergence of the construct studied: cognitive, behavioral, and affective. These 3 factors explained 57% of the total variance, an adequate percentage according to Pérez and Medrano [41].

Next, understanding that almost all the phenomena studied in the social sciences are interrelated and that the use of this type of rotation presents more interpretable structures, we opted for oblique rotation [42]. Thus, Table 2 presents the factor loadings after the use of this rotation. The structure was made up of 15 of the 32 initial items. Eight items were removed as they were grouped within a theoretical factor different from the proposed one, and 9 items with loadings lower than .50 were eliminated, following the recommendation of MacCallum et al. [37].

The resulting version was configured as follows: cognitive engagement (4 items), behavioral engagement (7 items), and affective engagement (4 items). The internal consistency according to ordinal alpha for this version was .91.

3.3. Confirmatory Factor Analysis

A CFA was performed to represent the relationships of latent variables with their observed variables and to verify that the items fit the proposed model [43]. To evaluate the goodness of fit of the model, the ULS estimation method and fit indicators were used. To evaluate the goodness of fit, indices less sensitive to sample size were used, such as the CFI and TLI, which yield an acceptable level of fit equal to or higher than recommended (>0.90), while the RMSEA is considered within the reasonable range (0.05–0.08) as an adequate model fit [44]. The chi-square statistic indicates that the model does not fit the data (, gl = 87, p=0.001). It is suggested to not consider this report, first, due to its sensitivity to the sample size (the larger the sample, the worse the obtained value), and second, it assumes that the model conforms to population parameters, which is very unlikely [45]. Table 3 presents the goodness-of-fit indices obtained.

It was also possible to determine that there are correlations between the factors. The correlation between the cognitive and behavioral factors was .64, between the cognitive and affective factors .53, and between the behavioral and affective factors .45. The factor saturations for the cognitive dimension ranged between .61 and .76, the behavioral dimension between .41 and .80, and the affective dimension between .66 and .84. Regarding internal consistency, according to ordinal alpha, the final version of the Comp-TA was .92, an adequate value according to Miranda-Zapata et al. [46].

4. Discussion

The main objective of the study was to validate the factor structure of the Comp-TA, an instrument constructed to assess the engagement to academic tasks of high-school students.

For this, specific goals were set: in the first stage, to design the questionnaire and in the second stage, to validate it through specific statistical analyses. Regarding the design phase, an instrument of 27 items was obtained, after they were positive-valued by experts. In relation to the validation phase, the results of the factor analyses supported the hypothesized structure of the Comp-TA for the three dimensions of engagement following the theory reviewed [14]. According to these researchers, academic engagement is formed by three dimensions: cognitive, behavioral, and affective. This appreciation is reaffirmed by Alrashidi et al. [18] who conducted a review of the use, definition, and conceptual dimensions of this construct. Thus, they present a variety of theoretical positions and conclude that Fredricks’ model has been essential to understand the phenomenon and has received the most attention in terms of validation and empirical examination.

However, two-dimensional and even four-factor models have been used. The rationale for two-dimensional models lies in the fact that behavioral and cognitive engagement have the effort indicator in common [13]. This can be glimpsed in the level of correlation between the two dimensions obtained in the factor analysis of the Comp-TA. Christenson and Reschly [23] mention that engagement, minimally, has a participatory behavior and some affective component. However, cognitive engagement, like affective engagement, are internal processes, unlike behavioral engagement. Given their nature, it is suggested not to use the two dimensions as one.

Concerning four-dimensional models, these consider the three dimensions of Comp-TA (cognitive, behavioral, and affective), and a new one is added: agentic. However, this new dimension requires further research to validate this proposal. At the same time, its conceptualization needs to be deepened due to its similarity to the behavioral dimension [18].

The resulting three-factor model of Comp-TA is consistent with the literature consulted. Likewise, it is empirically consistent with other instruments, which indicate that, although the factors of engagement are distinct processes despite being related, they can make contributions separately [12]. Thus, it is possible to analyze the implications of each of them on a specific academic variable, in this scenario, academic achievement.

Another relevant aspect, in the reliability analysis according to an ordinal alpha, is that our questionnaire presents an internal consistency of 0.92. The choice of this estimator over Cronbach’s alpha lies in the use of the polychoric matrix instead of the covariance matrix and given the ordinal nature of the data [38]. This unbiased estimator of reliability is barely applied, and its explanation could be the accessibility of the procedure, since other software does not implement it in its repertoire. Consequently, the Comp-TA differs from other instruments, as they report internal consistency through Cronbach’s alpha [13, 25, 27, 31].

The instrument proposed is different from the other proposals, since it focuses on the Chilean context. It is important to mention that the national educational system is mainly focused on a traditional culture of teaching and learning where academic tasks are privileged (e.g., individually and group homework), mostly outside the classroom. This pedagogical strategy allows students to complement their learning process in a more autonomous way, even if it is supported by the parents or tutors’ help. Consequently, the instrument takes on meaning and value, since it helps teachers monitoring the students’ engagement to his or her tasks, as well as to analyze his or her own pedagogical praxis.

Therefore, the resulting instrument evinced as a particularity a three factors’ model in accordance with the existing theory. In addition, the ordinal alpha is considered an indicator of reliability and it is focused on one particular aspect of the students’ actions: the academic tasks.

5. Conclusions

The Comp-TA, as a result of a rigorous design and validation process, is considered a valid and reliable instrument. First, its easy and quick application allows the measurement of engagement to academic tasks as a generic constructor in a differentiated way in allusion to its three dimensions: affective, behavioral, and cognitive. Second, it allows the study of the engagement to academic tasks in high-school students, since a representative sample based on the guidelines of the literature was used. Nevertheless, we consider it is possible to use this questionnaire at any higher educational level, since the items that compose it are easy to understand. Third, its use offers the opportunity to detect additional or opportune support for those students who gradually lose interest during in-class tasks and who in the future could drop out of the regular school system. Fourth, this questionnaire allows predicting students’ possibilities of continuing higher education studies when it is considering along other variables, such as academic achievement. Fifth, it allows teachers having feedback to review and improve their pedagogical practice. Sixth, it can be useful to evaluate the impact of programs that include innovative strategies such as gamification, role-playing, inverted classroom, and immersive technology, among others. In this sense, the type of instruction may or may not offer opportunities to promote student engagement in learning.

Overall, the Comp-TA fills a need for both teachers and researchers, given the lack of instruments to measure certain characteristics of students as behavioral and psychosocial aspects of the school context, proving to be useful for the knowledge of the relationship between engagement to academic tasks and variables such as academic achievement, motivational classroom environment, school satisfaction, and cognitive load, among others.

Guidelines are provided for the Comp-TA application and analysis as follows.

When investigating the generic construct of engagement to academic tasks, the scores are obtained by adding the values achieved for each item, following the additive nature of the instrument. The minimum possible score is 15, and the maximum is 105. The degree of engagement is distributed in 3 levels: low (15–35), medium (36–70), and high (71–105).

When studying each construct, the scores are obtained by adding the values for each item in allusion to the dimension.

The affective dimension is made up of 4 items: they make me curious to learn new things; they make me interested in studying new topics; they make me want to start a new topic; and I think they are interesting and attractive. The minimum possible score is 4, and the maximum is 28. The degree of affective engagement is distributed in 3 levels: low (4–12), medium (13–20), and high (21–28).

The behavioral dimension is made up of 7 items: I follow the instructions given by the teacher for its development; I answer the questions asked by the teacher; I complete all the requirements of it; I concentrate when the teacher presents the instructions to carry it out; I ask questions to the teacher when I do not understand; I participate in the revision of the work; and I listen attentively to the teacher’s explanations to solve it. The minimum possible score is 7, and the maximum is 49. The degree of behavioral engagement is categorized in 3 levels: low (7–21), medium (22–35), and high (36–49).

The cognitive dimension is made up by 4 items: I think about what I already know about the topic because it can help me understand better; I take care to understand as much as possible when reviewing the activity; I try to distinguish the most important information; and I think of different ways to solve it to choose the option I consider the best. The minimum possible score is 4, and the maximum is 28. The degree of cognitive engagement is distributed in 3 levels: low (4–12), medium (13–20), and high (21–28).

Finally, among the limitations of the Comp-TA, it should be mentioned that, despite its relevance and usefulness, the data are subjective in nature, and its accuracy will depend largely on the honesty of the student. As researchers and teachers, it is known that people tend to answer what is socially correct, especially in the academic context. That is why, we encourage to socialize the aims of the questionnaire prior to its application and also involving students in concrete actions to improve their level of engagement to academic tasks.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Ethical Approval

This study was approved by the Universidad Católica de la Santísima Concepción’s Ethics Committee.

Disclosure

This manuscript has not been published previously, it is not under consideration for publication elsewhere, and its publication is approved by the two authors.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Authors’ Contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Acknowledgments

This work was supported by the National Fund for Scientific and Technological Development, Fondecyt Regular (project no. 1191891 Integración de tecnologias inmersivas en educación: Mecanismos de aprendizaje y practicas educativas efectivas desde la formación de profesores), granted to Maria Graciela Badilla-Quintana by the Gobierno de Chile.