Abstract
One of the challenges in e-learning is the customization of the learning environment to avoid learners’ failures. This paper proposes a Stacked Generalization for Failure Prediction (SGFP) model to improve students’ results. The SGFP model mixes three ensemble learning classifiers, namely, Light Gradient Boosting Machine (LGBM), eXtreme Gradient Boosting machine (XGB), and Random Forest (RF), using a Multilayer Perceptron (MLP). In fact, the model relies on high-quality training and testing datasets that are collected automatically from the analytic reports of the Blackboard Learning Management System (i.e., analytic for learn (A4L) and full grade center (FGC) modules. The SGFP algorithm was validated using heterogeneous data reflecting students’ interactivity degrees, educational performance, and skills. The main output of SGFP is a classification of students into three performance-based classes (class A: above average, class B: average, class C: below average). To avoid failures, the SGFP model uses the Blackboard Adaptive Release tool to design three learning paths where students have to follow automatically according to the class they belong to. The SGFP model was compared to base classifiers (LGBM, XGB, and RF). The results show that the mean and median accuracies of SGFP are higher. Moreover, it correctly identified students’ classifications with a sensitivity average of 97.3% and a precision average of 97.2%. Furthermore, SGFP had the highest F1-score of 97.1%. In addition, the used meta-classifier MLP has more accuracy than other Artificial Neural Network (ANN) algorithms, with an average of 97.3%. Once learned, tested, and validated, SGFP was applied to students before the end of the first semester of the 2020-2021 academic year at the College of Computer Sciences at Umm al-Qura University. The findings showed a significant increase in student success rates (98.86%). The drop rate declines from 12% to 1.14% for students in class C, for whom more customized assessment steps and materials are provided. SGFP outcomes may be beneficial for higher educational institutions within fully online or blended learning schemas to reduce the failure rate and improve the performance of program curriculum outcomes, especially in pandemic situations.
1. Introduction
Today, the world is experiencing an unprecedented acceleration in the changing of life forms owing to the development of information and communication technologies (ICT). ICT continuously affects the social, economic, and cultural aspects of life. Many countries are trying to achieve success in devoting the information society [1] through infrastructure and policies to boost knowledge acquisition and become smart societies. In the academic domain, and owing to the rapid development of technological networks and smart devices, many Learning Management System (LMS) solutions have emerged [2].
During the Covid-19 pandemic, orientation toward e-learning and distance education has become necessary, and academic users are being more convinced about the importance of these new learning trends. In fact, universities worldwide have shifted from face to face-teaching mode to fully online learning with lockdown measurements imposed by governments and the need for social distancing. Thus, most educational institutions seek to rely on fully online or blended learning models to ensure educational process continuity [3]. Besides, the increasing use rate of the new learning and teaching models, the results show that many students are not familiar with this new environment and are stressed [4]. In fact, some studies mentioned that [5–9], and [10], increasing failure and dropout rates are observed. This is mainly due to unsuitable assessments and materials for students with learning and teaching difficulties. Today, the efficiency of online learning and teaching has become one of the most important research areas discussed in the educational community.
In this context, many researchers have concentrated their efforts on developing new solutions to address these challenges. The first researchers’ wave is oriented towards adaptive learning mode and recommended systems as solutions for these challenges, such as [7–12] and [13]. They state that adaptive education systems can significantly support students’ achievements via customized environments (i.e., teaching processes, materials, and assessments). Nevertheless, the quality of teaching processes and subject materials remains a significant challenge for these systems [14–17].
The second wave of researchers attempted to find solutions for predicting students’ failures. In fact, the rise of Artificial Intelligence (AI) has facilitated the development of a series of predictive models based on electronic online assessment and LMS tools. Faculty members are free to intervene and avoid student failures during learning, teaching, and assessment processes [5, 18–34].
Most of these studies use at most one predictive technique and have problems with performances and especially with low accuracy. Ensemble learning techniques are used as a solution for this issue, but it is necessary to consider the compromise between the number of systems used, the complexity, and the desired outcomes. Ensemble learning increases global accuracy by fusing the predictions of many learners. The most well-known learning techniques are bagging, boosting, and stacking. In this study, the ability of stacked generalization for classification is used to ensure customized teaching and learning tasks. The classification parameters can be the learner’s strengths, skills, interests, and needs. Each student’s class has a specific learning path to overcome learning weaknesses and guarantee success [27, 35–37].
In this study, a Stacked Generalization for Failure Prediction (SGFP) is used to enhance the classification performance of students and the prediction of their results. The proposed system allows dropout prediction and intervening in offering different learning paths to guarantee students’ success and reduce failure. Data is extracted from the Blackboard Learning Management System and each learning path is designed according to the students’ classes by the “Adaptive Release” Blackboard tool.
The organization of this paper is as follows: Section 2 presents the literature review where studies using ensemble learning techniques for prediction are presented and discussed. Section 3 explains the methodology of the study. Results analyses are detailed in Section 4, where we discuss the performance of our proposed approach. Section 5 describes the conclusion concerning this proposed system.
2. Literature Review
The effectiveness of online education is being debated once more, particularly during the pandemic era. Today’s educational institutions are expected to develop new educational solutions that are not bound by time or space. They need to create attractive learning environments and strategies to meet these expectations through e-learning, blended learning, mobile learning, and online education. Faculty members are also encouraged to create activities that allow students to actively explore and build their understanding of a given topic. They should succeed in their distance-learning courses. Furthermore, they must plan activities and learning tasks in a flexible manner while providing feedback to allow students to progress at their own accelerated rates [38–40]. E-learning systems concepts and recommendation methodology in the educational adaptation context have been the focus of several studies [7,8, 41].
In order to improve the quality of distance education, two main research areas of interest through LMS are identified: the first is concerned with adaptive learning and recommended systems, and the second is concerned with prediction of students’ performances.
2.1. Adaptive Learning and Recommended Systems
The first research line attempts to better understand students’ knowledge levels, as well as to recognize their preferences and how to learn and understand specific concepts based on students’ educational interests. References [9,15,42, 43].
In [10,11,44, 45], e-learning platforms are oriented to attract educational users and place electronic materials according to their preferences.
Shekapure and Patil [12] adopted a personalizing e-learning method that provides variable learning objects for students. The proposed method uses customized data, such as student knowledge and student learning style, to customize learning paths according to students’ profile classification. Indeed, the proposed approach is cognitive, progressive, and dynamic, where instructors can align e-learning material and sequencing to learners’ profiles and performances.
Cerna et al. [13] designed an innovative system that aims to impose an order in managing electronic course content accompanying topics related to the geography, history, and culture of English-speaking countries. Their proposed approach is a blended learning model that uses LMS tools for communication, navigation, and evaluation of student presentations.
Keskin et al. [41] discussed the individual factors represented in cognitive learning strategies: the extent of readiness for e-learning, and the motivation factor. The results of their study showed a correlation between learning environment preferences and the self-efficacy factor between e-learning motivation and task value, between the learning environment and self-efficacy constructs, as well as between e-learning motivation and task value. However, cognitive strategies, self-directed learning, student control, and anxiety factor testing were independent of students’ preferences for lecturing.
2.2. Prediction of Students’ Performances in E-Learning Environment
During the last few years, students’ performance assessment and improvement has been an important objective for all higher institution parties. This is in harmony with the workforce requirements of highly skilled and competent employees. Faculty members are expected to deploy all strategies, methods, and tools to motivate students to enhance their skills and competencies, and thus their grades, either in face-to-face or distance-learning environments. A new Educational Data Mining (EDM) focuses on predicting students’ performances in e-learning environments [46–49]. EDM applies supervised or unsupervised machine learning algorithms to inspect, analyze, and learn educational data, and then predict students’ performance.
The second research line introduced intelligent methods. Predicting students’ performance using statistical and machine learning techniques is no novel [43,50]. Although these works have been fruitful, applying the Artificial Neural Network (ANN) model is still broad for e-learning students’ performances compared to its use in other domains [51]. ANNs are imitations of human brain neuron functions to solve machine learning problems, and their use essentially leads to intelligent behavior. ANNs can be used to solve many types of problems such as forms, speech recognition, and function approximation, but their application in the eLearning field is especially based on classification and prediction [52–54].
Arsad et al. [55] presented an ANN-based system for early performance prediction. The study was conducted with engineering students at a Malaysian university. In fact, academic achievement in semester eight was measured using the cumulative grade point average. Moreover, Adewale et al. [56] used a feedforward neural network applied to secondary school students to study the relationship between cognitive and psychological variables on academic performance. The authors concluded that clustering students according to their performance into different categories using ANN is an efficient method that enables curriculum developers and educational planners to provide better educational services. In [21], the authors utilized data mining to identify patterns and student grouping used to explain academic dropout. They collected data from students who registered for two admission periods at the Universidad Tecnologica Indoamerica in Ambato, Ecuador. They classified and defined the performance patterns using a k-means algorithm, and predictions for new students were made using a Support-Vector Machine (SVM) model. The study in [24] predicts grades and proposes a deep learning model comprising scattered attention layers, convolutional neural layers, and a fully connected layer. Grades, student demographics, and course descriptions were among the information gathered. The proposed model achieved 81% prediction accuracy and 85% failure prediction accuracy and provided a potential explanation for the predicted outcome. In [6], the study employs machine learning models to improve the prediction of previous student performance and explain why a student’s performance reaches a certain score. It also provides a visual technique to assist in determining the factors that most influence the score through the experiment, allowing educators to identify students at-risk early and provide appropriate exhortation in an advantageous way.
Furthermore, Kalyani et al. [36] applied a convolutional neural network (CNN) model to predict and assess student performance. The number of hours the student spent studying and the degree of student involvement in academic activities were used as predictor variables. In [37], the recurrent neural network (RNN) model was used to predict the final grade and was compared to a multiple regression analysis model, in which RNN was applied for early prediction of 108 students’ results and confirmed its effectiveness.
A comparison study was conducted in [57] to predict students’ academic performance based on a single performance factor. Both multilayer perceptron neural network (MPNN) and generalized regression neural network (GRNN) learning algorithms are employed on collected data from documents and student transcript records. The findings t reveal that the overall performance of GRNN outperforms the MPNN Multilayer Perceptron model with an accuracy of 95%. In addition, the study concluded that GRNN could be used by educators to predict student academic performance based on a single performance factor.
Although the mentioned models provide a high capacity for classification and prediction of student failures, they rely on individual classifiers that have little knowledge of the dataset. Hence, the use of the ensemble learning method increased the average prediction performance.
2.3. Ensemble Learning Approach for Prediction in E-Learning Environment
The stacked generalization model is one of the most well-known learning techniques. It combines many learners and uses their outcomes as an input to the meta-learner to predict the final student class. Previous studies have deployed the stacked model in various fields to increase the accuracy prediction and decrease the lowest prediction error. For instance, in the education field, the authors in [35] presented a stacked generalization model composed of three learners: back propagation neural networks, support vector machines, and M5P model tree. This study aims to predict academic achievement after graduating from students. The authors used the root mean square error (RMSE) to evaluate the model performance. The prediction result of stacking compared to the three classifiers was better. In [58], the authors presented a model that combines stacking and voting ensembles to evaluate faculty performance. They used two datasets: the first is from the UCI machine learning repository called the Teaching Assistant Evaluation; hence, the second is from the students of a university. They employed 15 algorithms. When compared to methods using only one model, the proposed method produced higher accuracy, but the problem of complexity arises because the use of 15 algorithms generates a delay in execution; hence, there is a need to respect the compromise between the desired results and the complexity.
Alizamar et al. [59] presented a stacked RASCH model to represent the differences in aggressive behavior between male and female students at the Junior High School of West Sumatera.
The authors in [60] used ensemble learning to identify style learning based on the Vark model (visual, auditory, reading/writing, and kinesthetic). They have used J48, SVM, Random Forest, Nave Bayes, and hard majority voting. The results obtained in the 20% Test set are as follows: the J48 decision tree achieved 57.2% precision level and 61.4% accuracy, the SVM classifiers achieved 58.5% precision level and 59.6% accuracy. The Random Forest achieved 57.9% precision level and 60.5% accuracy. The Naïve Bayes achieved 59.7% precision level and 62.3% accuracy. The Hard Majority voting achieved 60.2% precision level and 62.9% accuracy.
In [63], the authors evaluated eight classification techniques in order to recognize the parameters that make a significant contribution to providing an excellent model for classifying a student based on his performance. The J48 Decision Tree classifier achieved 93.5% precision, 93.5% recall and 93.2% F1-score. The Logistic Regression achieved 91% precision, 89.6% recall and 90.3% F1-score. The Multi-Layer Perceptron achieved 92.5% precision, 90.5% recall and 91.2% F1-score. The Support Vector Machine achieved 96% precision, 89% recall and 92.4% F1-score. The AdaBoost (Adaptive Boosting classifier) achieved 90% precision, 85.9% recall and 92.4% F1-score. The Bagging achieved 96.9% precision, 92.4% recall and 91.8% F1-score. The Random Forest achieved 97% precision, 90.8% recall and 93.8% F1-score. The Voting achieved 93.1% precision, 91.4% recall and 92.3% F1-score.
In [64], the authors predicted student academic performance by proposing a Hybrid Ensemble Learning Algorithm (HELA). The Super Learner algorithm receives prediction results from base classifiers such as Gradient Boosting, Extreme Gradient Boosting, Light Gradient Boosting Machine, and various combinations of these algorithms. A Random Search algorithm is used to optimize the hyper-parameters of base classifiers. The proposed algorithm predicts students’ performance in two courses, and the experimental results achieved 96.6% and 91.2% accuracy values.
The study in [65] suggested an ensemble learning method based on label distribution estimation called light gradient boosting channel attention network (LGBCAN). This model is employed to forecast performance in web - based learning tasks. The Channel Attention Network (CAN) model enhances LightGBM’s function by concentrating on better outcomes in the K-fold cross entropy of LightGBM. The LGB achieved 68.33% accuracy, 68.33% precision, 59.03% recall and 61.58% F1-score. The XGB achieved 67.71% accuracy, 62.45% precision, 58.63% recall and 60.47% F1-score. The LGBALD achieved 57.84% accuracy, 63.73% precision, 57.14% recall and 60.26% F1 score. The XGBALD achieved 56.76% accuracy, 63.23% precision, 56.96% recall and 59.93% F1-score. The LGBCAN achieved 68.14% accuracy, 63.56% precision, 63.66% recall and 63.61% F1-score.
Likewise, most of the mentioned studies use datasets collected from questionnaires, surveys, student registration units, and students’ transcripts to train and test models [27, 36, 37, 55–58, 61].
The weakness of some studies is related to the potential issue with data quality, which is in some way outdated, inaccurate, subjective, and does not reflect the real students’ activities through e-learning processes. It is obvious that to better predict students’ failure and dropout, significant and objective data will be of great help. Indeed, learning algorithms/models’ results in e-learning environments are more interesting if data are directly used from LMS platforms.
To deal with these limitations, this study presents the idea of exploiting LMS solution analytical reports. In fact, all LMS platforms provide an amalgam of easy-to-use and complete analytical generating report tools. These reports are interesting sources of knowledge to investigate: (i) all students’ assessment grades are there, (ii) the content reflects the real students’ e-learning activities (i.e., access to course contents, assignment submissions, interactions, etc.), and (iii) the data is clean and well structured. However, to the best of our knowledge, this perception is still lacking.
This study focuses on offering the SGFP model to avoid student failures by exploiting LMS analytic reports for a specific set of courses and for a fixed academic year. The study was conducted to predict students’ performance and avoid dropout. The training and testing datasets are extracted automatically from Blackboard analytic reports through its Analytic for Learn (A4L) and Full Grade Center (FGC). The SGFP algorithm is based on heterogeneous data reflecting students’ interactivity degrees on one side and students’ educational performances and skills on the other. These data are more significant and better for the e-learning process and evaluation. Extracting updated data from real student activities offers accuracy, objectivity, and precision. SGFP uses the Blackboard adaptive release tool to customize learning paths for each student’s class.
3. Methodology
The research methodology of this study was based on the classification capacities of the stacked generalization. It is composed of six steps: (1) data collection, (2) data preprocessing, (3) SGFP modeling, (4) training and testing, (5) evaluation, and (6) validation.
3.1. Data Collection
The LMS creates a large amount of data. Universities are optimizing instructional design, increasing faculty effectiveness, and identifying at-risk students in time to keep them on track to graduate with high-quality credentials by deliberately presenting this data in the form of information. Like any LMS, Blackboard provides educators with a better way to access the information they need to assist and maintain their students’ progress. Instructors can access course dashboards and other reports from Blackboard A4L directly from Blackboard Learn courses. All students enrolled in the class are listed in Table 1: T: theoretical course; P: practical course).
The data extracted included 376 students enrolled from the start of the 2019-2020 academic year to the end of the first semester of the 2020-2021 academic year. Figure 1 shows SGFP training and testing data extracted from Blackboard for 22 courses from the Information Sciences Department (ISD) at the College of Computer and Information System (CCIS) and the Computer Science Department (CSD) at the First Common Year College (FCYC).
3.1.1. Input Data
Table 2lists the input and output of the SGFP dataset. The dataset was collected via Blackboard tools as follows:
A4L: This allows instructors to keep track of how students are performing through running course reports. A4L extracts data from the UQU system information science (SIS) and Blackboard-Learn. Four types of reports are provided from A4L: Course at-a-Glance, activity grade scatter plot, activity matrix, and course submission summary report. The SGFP classification is based on six predictor variables: five variables from the Blackboard-Course at-a-Glance report:(i)Access Operations (AO): this indicator refers to the number of operations performed by the student.((i)LMS time spent (LTS): this metric indicates how long a student spends per week browsing the online course on Blackboard.((ii)Degree of student interactions (DSI): this indicator measures during the week, the degree of
Student interaction with the course and its tools, interactions with the teacher, and interactions with his colleagues.((i)School Assignment Submission (SAS): this indicator reflects students’ commitment to homework.and assessment tools.((ii)Grade of Evaluation Center (GEC): this parameter represents the grade obtained by students during subsequent participation through Blackboard.
FGC: this is a container for all students’ assessment grades, and the sixth input variable, Grade of Initial Exercise (GIE), was extracted from FGC.
The SGFP model entry matrix is composed of seven columns (six columns for input and one column for output) and 376 rows. Equal low weights (12%) were assigned to the five first inputs, and the highest weight (40%) was assigned to the sixth variable (GIEe). On the one hand, the rationale for the weights associated with these parameters is that students’ activities on the LMS platform cannot always be meaningful.
Some students can hardly navigate the LMS, download course content, work seriously offline, and achieve good results. Therefore, the highest weight is for the initial exercise evaluation as it more reflects the students’ performance and is decisive to e-learning course customization. On the other hand, equitable weighting was chosen for the five A4L indicators because their impact on student rankings is similar, and they represent students’ interest in digital education and measure their virtual interactivity with online courses, but they do not reflect their actual scientific performance. Weak weights are given because the parameters that represent the impact of browsing time, operations performed, interactions with LMS content. All have the same degree of importance in evaluating student performance.
3.1.2. Output Data
Students’ profiles are classified into three classes (A, B, and C) according to the six predictor variables. For each class, a special learning path is fixed as follows: For class A, students were allowed to take the exam directly without taking any other support courses. Their learning path offers video sequences containing the essential summaries of the courses. For class B, students with grade above average might present a risk of not obtaining 60% in the final exam. The learning path offered these students descriptive summaries and short-corrected tests covering all course modules. For class C, to avoid dropout, a customized learning path is then offered by summing up the descriptive summaries and recapitulative units. Each unit ends with an evaluation test that is essential for students to move from one unit to another. The serious application of this learning-path customization can allow for an excellent success rate.
The Blackboard Adaptive Release tool was used to customize learning paths. Adaptive Release is a dedicated publishing tool that allows instructors to deliver personalized content to students. This content is based on a set of rules related to four criteria: date, membership, grade, and review status. The goal of personalizing the publication of content is to create courses that are more interactive and tailored to the needs of each learner.
3.2. Data Preprocessing
SGFP learning data were extracted from the Blackboard Course at-a-Glance report and from the FGC tool one month before the final exam (Figure 2). Then, they were normalized according to the weights assigned to them. These variables will then be put in an Excel file. The columns of this file represent the five normalized variables extracted from A4L, while the rows represent the students’ names.
The sixth input variable was taken from the FGC. This variable must be normalized according to the weight assigned to it, then it will be placed in the sixth column of the Excel file.
Regarding SGFP targets, the data are extracted from the Blackboard-FGC tool, and simple programming in Excel allows the organization of this variable, which is the seventh column of the Excel file. Three values were used for this response variable. The data placed in order in the Excel file were read by our proposed model to ensure the learning stage.
All data were extracted from Blackboard and computed in the same manner. Blackboard determines the highest level of activity within a given class and course, and then computes the mean for each parameter. All students in the same class will be ranked in relation to the average, with a green arrow pointing up if they are above the mean value, and a red arrow pointing down if they are below the mean value. After assigning a weight to each parameter, we proceeded in a standard manner to normalize these variables, as
3.3. SGFP Modeling
Stacked generalization or stacking is an ensemble learning technique that is used to fuse diverse machine-learning algorithms.
This is a hierarchy of learners’ levels. More specifically, the stacking is contained in two levels: the first is composed of the base learners, while the second is composed of a meta-learner. The main idea behind the stacked generalization, besides the combination of the prediction values from the base learners, is to improve the prediction performance over all the learners in the ensemble. The stacked method was employed for failure prediction. In this work, four different learners named Random Forest (RF), XGBoost (XGB), Light Gradient Boosted Machine (LGBM), and Multilayer Perceptron (MLP), were used to form a stacking model. All the first-level classifiers of our model are ensemble learning models based on decision tree algorithms.
3.3.1. Base Learners
(i)Random Forest (RF) is an ensemble method built on decision trees. Indeed, it comprises many trees. Each tree in the forest provides a classification, known as tree votes. The forest fusions all vote trees and chooses the highest vote as the final prediction. Random forest is an efficient method and is highly accurate because of the number of decision trees used in the process.(ii)Extreme Gradient Boosting (XGB) is based on the principles of the gradient-boosting framework. It is a tree ensemble approach in which the trees are added sequentially, and each tree learns from its predecessors, where they aim to minimize the errors of the previous tree. The trees are provided in parallel tree boosting to solve tasks quickly and accurately. XGboost is created to thrust the extreme computational limits of machines to provide a scalable and efficient library.(iii)Light Gradient Boosted Machine (LGBM) is a decision tree ensemble method for regression and classification tasks. It divides the tree leafwise with the best fit. LGBM reduces the loss of the level-wise algorithm when it is growing on the same leaf, which makes the predictions more accurate. LGBM is a fast, efficient, and distributed gradient-boosting framework.
3.3.2. Metalearner
(i)Multilayer Perceptron (MLP) is a type of feedforward artificial neural network. It is inspired by the sophisticated functionality of human brains, where there are hundreds of billions of interconnected neurons. The MLP includes three layers: the input layer, hidden layer, and output layer. MLP works using the backpropagation technique for trains.
3.4. SGFP Parameters
The specific parameter settings of different machine learning algorithms are presented in Table 3.
3.5. SGFP Steps
The algorithm below describe the process of the SGFP model in this work:
| ||||||||||||||||||||||||||||||||||||||||
The framework of the proposed method is shown in Figure 3. The groups of students were divided for each section into two subgroups: the first group represented 90% of the students reserved for learning, and the second group was reserved for testing the model. The K-fold cross-validation technique is used to avoid the use of the same training dataset at both method levels, particularly to avoid overfitting.
In the first step, the divided dataset was used to train and test the first-level base learners of stacking. In the next step, the meta-dataset is generated from the predictions of the first-level models. Then, the MLP is trained and tested using the constructed feature instances to produce the student class. Students are classified into three groups according to their final marks, as shown in Table 4 and Figure 4.(i)Class A: student’s grade between 80 and 100(ii)Class B: student grades between 60 and 80.(iii)Class C: student grades less than 60.(iv)The input vectors and target vectors were randomly divided into two sets as follows:(v)90% were used for training.(vi)The remaining 10% was used as a completely independent test of the SGFP model.
3.6. SGFP Evaluation
To evaluate our proposed SGFP model at different levels, it was compared with the employed base classifiers in terms of accuracy, precision, and recall. For given input data, a binary classifier generates output with two class values 1/0. The one of interest is typically represented as “positive,” while the other is denoted as “negative.” The observed labels for all data instances are contained in a test dataset used for performance evaluation. Having followed classification, the observed labels are compared to the predicted labels to determine performance. If a binary classifier’s performance is perfect, the predicted labels will be identical, but it is relatively rare to be able to develop an ideal binary classifier that is useful in a variety of situations.
The confusion matrix is constructed from the three components of binary classification. A binary classifier predicts whether all data instances in a test dataset are positively or negatively. True positive, true negative, false positive, and false negative are the four outcomes of this classification.(i)True positive (TP): correct positive prediction(ii)False positive (FP): incorrect positive prediction(iii)True negative (TN): correct negative prediction(iv)False negative (FN): incorrect negative prediction
Accuracy is the percentage of correct predictions that a learner has achieved. It is computed by dividing the number of correct estimates by the total number of predictions: (i)Precision, also known as the positive predictive value, is the ratio of the pertinent instances to the retrieved instances:(i)Recall, also called sensitivity, is a fragment of the retrieved relevant instances:(i)The F1-score is a statistical measure that combines precision and recall with rate performance.
As shown in Figure 5 and Table 5, the accuracies of all the base classifiers excluding the LGBM model vary when dealing with three test samples resulting from the 10-fold cross-validation technique.
Although they are some classification problems, it is obvious that some methods perform better in one and not so well in the other. However, the SGFP method exhibited consistent and high accuracy. Moreover, the mean and median accuracies for the SGFP model are slightly higher than those of the other classifiers.
To demonstrate the advantage of the choice of the MLP model as a meta-classifier, it was compared with other models in terms of accuracy. It is obvious that the MLP algorithm outperforms the other algorithms, with an average accuracy of 97.3%.
Table 6 shows that our model successfully recognized the three student classes with a sensitivity average of 97.3% and that the prediction is valid when the relevant class is estimated with a precision average of 97.2%. Moreover, our model had the highest F1-score (97.1%).
This was expected, given that the proposed technique is built on a combination of an ensemble of failure prediction models rather than predictions of single learners (LGBM, XGB, and RF).
4. SGFP Validation
Having learned and tested, the proposed model can classify students into three classes. The validation step of the SGFP was carried out on 176 students who were studied during the academic year 2020-2021. The data retrieved from Blackboard were for the courses listed in Table 7.
The ranking results will be taken into consideration to guide students towards educational paths. To guarantee this mission, Blackboard’s “Adaptive Release” tool is used to enforce the follow-up of the paths traced for the students. After training the SGFP model, the program was ready to automatically classify every new student.
The student thus classified in a very specific group would be obliged to follow an educational path before taking the final exam. Blackboard provides a dedicated publishing tool called “Adaptive Release” that allows instructors to deliver personalized content to students. This content is based on a set of rules created by the instructor. The rules are related to four criteria: date, membership, grade, and review status.
Figure 6 depicts three flowcharts that correspond to three classes of students. Each student will be able to automatically follow a learning path and this according to the class he is belonging to.
The goal of personalizing the publication of content is also to create courses that are more interactive and tailored to the needs of each student. Blackboard offers two types of adaptive versions: basic and advanced. For the first type, instructors apply only one rule to a piece of content for all criteria types. Students are required to meet all the rules’ criteria before the item (e.g., file, image, video) is published. Students belonging to class C had to follow a complicated path with specific criteria and item output restrictions. It is obvious that the path course for class C students is more difficult to access. The second type of “Adaptive Release” is advanced, and instructors can set more complex criteria. They can add different options and criteria to rules. Students must meet all criteria in one of the rules to access them. Ten sections were applied during the first semester of the 2020-2021 academic year. The results proved their effectiveness.
5. Results Analysis
The application of the SGFP approach on 10 sections during the first semester of the academic year 2020-2021 raised the following points (Table 8):
Before sitting on the midterm exam, the students were classified as follows: 28% in class A, 60% in class B, and 12% in class C. This means that a significant percentage of students were threatened by failure.
Three learning paths were customized to each students’ class, and the results of the classification after passing the midterm exam were 36% in class A and 62.86% in class B, while class C contained only 1.14% (only two students).
5.1. SGFP Performances
The experiment of the SGFP algorithm on 10 sections of 176 students enrolled during the academic year 2020-2021, allowed the success of 174 students and the failure of only two; so, 98,86% of the students took the midterm exam. Moreover, the SGFP approach gave satisfactory results when tested with additional students enrolled in different colleges in Saudi universities (such as, college of nursing, college of business). Prior to the exam, instructors extract data from the Blackboard (Evaluation -course analytics-) via the ‘Course at-a-Glance’ option to generate input data for the SGFP approach, and all enrolled students are classified. Then, instructors confirm the testing and validation of the model. The particularity of this method lies in the fact that adequate learning of the SGFP model allows for an efficient classification of the student. The results show that the performance of SGFP is almost perfect. All the learning paths taken by the students were customized with the “Adaptive Release” tool of Blackboard and this according to the obtained grades in subject assessments. In fact, students were not able to move from one item to another only if they passed the test with a score of more than 70 percent. The Shareable Content Object Reference Model (SCORM) package is used to generate an individual learning path in the LMS (i.e., personalized contents navigating and sequence, assessments).
The results show SGFP approach performs better than one classifier learning model. F1-score, accuracy and precision values are higher compared to LGBM, XGB, and RF models.
5.2. Limitations
Our promising SGFP results are data dependent. The dataset used for learning and testing has the distinction of being objective, clean, and reflective of student behaviour in the online environment. So, we suspect two major limitations related to:(i)Training/testing corpus size: using only one testing and two training semesters is not enough to maximize the SGFP performances. The recommendation here is to increase the number of the semesters but without overfitting the model.(ii)Dataset source: our approach uses LMS students’ data. Even some students feel to be more confident to surveys data. The limitation here is that data related to students’ behaviour can be interesting to consider. Thereby, an amalgamation of survey data and LMS data could increase precision and accuracy of SGFP approach.(iii)Exhaustiveness of attributes: here we use student university data encompassing only traditional online assessments (Access Operations, LMS Spent Time, Grade of Evaluation Center, etc.). We think this is not sufficient, and non-traditional attributes can be added to predict students’ failure in online environment (i.e., students engagements variables, attendance, time point intervention, etc.). A comprehensive study including these variables will be considered in future work by adding more data from LMS platforms (i.e., students files logs, students’ participations on discussion forums, blogs, wikis, etc.).
6. Conclusion
In this paper, a stacked generalisation-based algorithm (SGFP) is proposed to predict and avoid student failures using data from the analytical reports of learning management systems and grade containers of undergraduate students.
The robustness of this study is linked to three factors:(i)The data used to classify the students are overly significant(ii)The classification method is based on three prediction models rather than being content with a single model(iii)The automatic design of customized learning paths for each student depending on the class he belongs using LMS tools and this without intimidating the student
The first stage of this study was the classification of students according to their performance using data from A4L and FGC. The authors believe that to better predict student dropout, meaningful and objective data will be of great help. The SGFP algorithm is based on heterogeneous data that reflects students’ levels of interactivity and their performance and teaching skills. These data are more relevant and beneficial to the e-learning process and assessment. Accuracy, objectivity, and precision were obtained by extracting updated data from actual student activities. Indeed, the results of algorithms/learning models in e-learning environments are more fruitful if the data used comes directly from LMS platforms.
The findings are three students’ classes (Class A: above average, Class B: average, Class C: below average). The second stage is to customize learning paths (contents and assessments) according to students’ classes by using the “Adaptive Release” tool capabilities of Blackboard. This tool is used to personalize learning routes and is based on a set of rules regarding four different types of criteria: date, membership, grade, and review status. The customized content publication also aims to create courses that are more dynamic and appropriate for the needs of each learner.
The principal goal achieved is the improvement of students’ success rate (98.86%). After using the SGFP model, students in class C will almost certainly fail (some students in class B are threatened by failure). The results show a significant improvement in students’ success rates in the academic year 2020-2021. For class C students, the failure rate increases from 12% to 1.14%, for whom more evaluation steps and contents are customized. Following well-defined learning paths, only two out of 176 students failed. Therefore, the success rate was 98.86%.
The SGFP approach, which is built on a combination of an ensemble of failure prediction models, identified students’ classes with a sensitivity average of 97.3% and a precision average of 97.2%. Compared to base classifiers, the results show that the mean and median accuracies of SGFP are higher. SGFP had the highest F1-score of 97.1%.
The robustness of this study to better predict failure is primarily linked to the good quality of the data, which are recent, significant, and objective. The learning phase from indicators extracted from e-learning environments is very effective because the data used comes directly from LMS platforms. This study comes to face the limitations of the previous works by presenting the idea of exploiting the analytical reports of the LMS which records all the activities of the students in detail and represents a clear mirror of each student. In fact, all LMS platforms provide an amalgamation of comprehensive and easy-to-use analytical reporting tools. These reports are interesting sources of knowledge to explore in adaptive learning. Concerning the classification robustness, the three models chosen had presented their satisfaction after several tests.
Among the recent publications, we notice that our study presented a clear improvement. Indeed, the precision of SGFP is 97.2%, while the best precision value obtained from [60] is 60.2%, achieved by the Hard-major classifier. The study in [62] reached 97% with the random forest classifier. A precision of 63.73% was achieved by LGBALD classifier in [63].
8F1-score achieved by SGFP is 97.1%, while the best value obtained for the published recent works in [26, 60, 62, 63] is 93.2% achieved by the J48 Decision Tree classifier.
Finally, the outcomes obtained in this study can be beneficial for higher institutions implementing a full distance-learning mode or in some pandemic health circumstances to improve student’s performance and hiring rates in local and internal labor markets. This work is still open for improvements by (i) enhancing the stacked generalization training performances by combining rule-based classification and learning datasets, (ii) implementing adaptive learning according to learners’ preferences and learning styles (i.e., visual learners, social learners, auditory learners, etc.), and (ii) adopting other classification algorithms and models and evaluating and comparing their results [17, 61, 64, 65].
6.1. Ethic Statement
This study did not involve human participants; there was no need for participant consent in this study, and no minors were included in this study. This study did not report medical records or archived samples.
Data Availability
The data used to support the findings of this study are available from the author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The authors would like to thank the Deanship of Scientific Research at Umm al-Qura University for supporting this work by grant code 18-EDU-1-01–0004.