Abstract
During the recent epidemic period of COVID-19, online courses have become an important learning form for college students. However, online learning cannot communicate face to face in class and position students’ abilities accurately, and there are many problems and limitations such as one-way evaluation, for example, neglecting of process evaluation and simple evaluation form. Therefore, how to construct the evaluation system of online course teaching and make effective use of the evaluation mechanism becomes an urgent problem. Based on the big data mining of online course evaluation data, the online course evaluation optimization architecture based on process evaluation is proposed. The optimization of online course evaluation is analyzed from online course evaluation data and student comments using deep learning and collaborative filtering technology. This includes improving teacher teaching and improving student learning efficiency. Data experiment proves that the proposed algorithm can provide an optimal evaluation strategy, guarantee the students’ learning quality, and improve the efficiency of online course.
1. Introduction
The outbreak of COVID-19 has led to the shift of courses from live to online. Online courses are available in live broadcast and on-demand mode. Compared to traditional classes, the feedback of online courses is provided in the form of comments and posts. Although online courses solve the problem of students’ listening, the learning quality is not satisfactory, and many problems occur. For example, online courses are difficult to understand, and the teaching effect is difficult to guarantee. Specifically, most colleges are vulnerable to formal influences when developing online courses, which may change the teaching content at will. As a result, the teaching content is not properly correlated, and a complete knowledge system cannot be formed. In addition, the teaching method is focused on the teacher-oriented one-way inculcation. The student’s main role in the classroom is not fully reflected and the learning efficiency is low. Teachers also have difficulty getting accurate students’ feedback. Furthermore, limited by teaching form, there is a lack of effective classroom discussion among students.
According to the above observation, one of the root causes of online course problems is that teachers cannot effectively obtain student’s feedback during course development, which results in information asymmetry. Fortunately, in just over a year (during COVID-19), online courses have generated a lot of education big data. Education big data is the sum of data generated during the teaching and receiving of knowledge. According to Gartner report [1], big data is a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain period of time; it is a massive, high-growth, and diversified information asset that requires new processing models to have stronger decision-making, insight, and process optimization capabilities. IBM proposes that big data has “5V” characteristics: Volume (large amount), Velocity (high speed), Variety, Value (low-value density), and Veracity. Compared with traditional big data, education big data has features of two-way feedback, changeability, extensiveness, and specificity.
Education big data can reshape the three major characteristics of learning: feedback, individualization, and probabilistic predictions [2]. We believe that education big data mining can collect students’ feedback data on curriculum resources, implement personalized online courses resources to meet students’ individual needs, and optimize the learning content, time, and learning mode of online courses resources through probability prediction to incorporate empirical data into online courses resources. With the support of education big data, information asymmetry in online courses is broken.
Data mining refers to the process of searching for hidden information in a large amount of data by using a specific algorithm. Essentially, data mining is an information search technology based on data management architecture and searching algorithm. Aimed at optimizing online courses, we propose a Feedback Mining-based Online Course (FMOC) optimization architecture in the context of education big data. FMOC is a teaching mode supported by education big data. As depicted in Figure 1, it includes teaching, differentiated feedback, guidance, and closed-loop feedback for course evaluation optimization, enabling a virtuous cycle of teaching activities. This provides a theoretical basis for studying the optimization of educational and teaching curriculum resources at other levels. In addition, based on FMOC, we propose an optimization scheme of online course teaching evaluation system and systematically analyze and evaluate the effect of the scheme. Then, based on the evaluation, a personalized exercise recommendation method combined with Deep Knowledge Tracing is proposed to provide suitable test questions for students. Finally, we synthesize all the feedback to form a personalized curriculum opinion model to promote the teacher to improve the online courses.
The rest of the paper is organized as follows. Section 2 provides an analysis of related work. In Section 3, the evaluation and optimization scheme of online course learning based on formative evaluation is proposed. Section 4 presents a personalized exercise recommendation method based on knowledge tracing. Section 5 describes the individualized opinion model for teacher improvement. The effectiveness of the scheme is evaluated by simulation experiment in Section 6. Finally, conclusions are drawn in Section 7.
2. Related Work
Traditional course evaluation focuses on students’ final academic achievement and ignores the learning process. The formative evaluation shifts the focus to the students’ learning process, for example, the actions, effects, and willingness in the process. It can help students correct their learning habits, adjust learning methods and objectives, and detect their learning effects and finally achieves the goal of completing the course and improving students’ ability. Reference [3] proposes the theory of formative evaluation and final evaluation and their difference. The theory has greatly influenced the education circle, especially in language and program language teaching. Reference [4] proposes that teaching evaluation can be divided into three categories: diagnostic evaluation, formative evaluation, and final evaluation. The three kinds of evaluations have different evaluation objectives and will produce different evaluation results. Diagnostic evaluation [5] is generally used for the stage when a course is just identified and emerging.S
The existing problem recommendation methods are mainly divided into the method based on collaborative filtering [6] and the method based on knowledge modeling [7]. The research works are as follows: [8] generates personalized recommendations and finds other users with similar interests by submitting comments on educational resources and visiting and searching for comments from others; [9] uses collaborative filtering and data mining technologies to analyze students’ reading data and generate recommended scores to help students select appropriate courses. Reference [10] proposes the student personalized sorting algorithm EduRank, which uses collaborative filtering to find similar students and summarizes the rankings of similar students to construct the problem difficulty ranking of the target students. The algorithm can be used to assist teachers in customizing exercise sets and exams for students. Reference [11] proposes a recommendation module Protus of a programming tutoring system, which uses the Apriori algorithm to mine frequent sequences to analyze the habits and interests of similar students and makes personalized recommendations based on the ratings of these frequent sequences, which can automatically adapt to students’ interests and knowledge level. Reference [12] proposes a learning material recommendation method based on fuzzy tree matching.
Corbett et al. propose Bayesian Knowledge Tracing (BKT) [13]. BKT models students’ underlying knowledge state into a set of binary variables, each of which represents the understanding or lack of understanding of a single concept, using the implicit Markov model, and iteratively updates guessing rate, learning rate, error rate, and learning initial probability of each knowledge point to obtain students’ knowledge point. Reference [14] introduces students’ learning speed parameters into BKT, which increases the accuracy of BKT. It is also used in the intelligent tutoring system to guide students in learning. Deep Knowledge Tracing (DKT) [15] was proposed to train RNN to model students’ knowledge status. DKT achieves better effects than BKT without manual labeling of the training dataset. In recent years, many scholars have applied and improved DKT: [16] proposes more features for learning and adding automatic encoder layer to reduce the input dimension and improve the model performance. Reference [17] uses bidirectional LSTM to train the text representation information of the question and introduces the attention mechanism in the process of knowledge state modeling to improve the accuracy of knowledge tracing by combining the two models. Reference [16] reconstructs the loss function of the DKT model and adds three regular term parameters to improve the stability and accuracy of the mode. Reference [18] encodes students’ programming submission records into the DKT, trains the model, and predicts students’ performance in subsequent programming tests.
3. Online Course Evaluation Scheme Based on Process Evaluation
Process evaluation (PE) emphasizes timely information feedback to help teachers revise and improve their curriculum, thus improving teaching and learning effectiveness. Thus, the tasks involved in the process evaluation include the following:(1)Tracking the learning process: To track the learning process, the teacher is required to assign different learning tasks at each learning stage, clearly describe learning objectives, and design evaluation methods for these objectives. To achieve their learning goals, students will adjust their learning methods based on the evaluation results.(2)Testing the learning effect: The formative evaluation should give students a clear understanding of whether they have completed the current stage and what they are going to do for the next stage. This helps to enhance students’ interest in learning, enhance students’ confidence in learning, and improve their learning methods.(3)Identifying problems: Based on feedback and analysis of previous comments, both teachers and students can find out the causes of problems through formative evaluation and seek the best solution to the problem.(4)Correcting student learning strategies: Teachers can contact students in a timely manner and work with students to build and improve learning strategies and methods based on problems in the learning process. This process will greatly improve students’ potential abilities.
Based on the above steps, the process of evaluating and optimizing the online course learning process is shown in Figure 2. The proposed PE can track the whole process of online learning and record and check all learning processes.
In order to better use data mining technology, weighting coefficients are considered, the classification of variable continuation (Figure 2) is refined, and the evaluation optimization method is modeled from the following aspects.
AHP [19] is a system analysis method proposed by T. L. Satty. The basic principles of AHP are as follows: a set of variables that contain specific factors related to a complex problem. It is modeled based on a composite structure model, and factors at each level are compared and ranked by importance. Finally, the weight of each influence factor in the decision-making problem is determined.
Firstly, set a decision matrix, sort each comparison row, and generate a decision matrix , where is the total number of associated pairs of comments. The evaluation weights are then confirmed by AHP. Solve the eigenvalue for the rank of the decision matrix to obtain the maximum eigenvalue and the corresponding eigenvector . The weight calculation results are sorted at different levels based on the sequence of feature values. Finally, the consistency check of the decision matrix needs to calculate the consistency index . The random consistency ratio meets the following requirements: . It is considered that the decision matrix meets the requirement of consistency. Otherwise, the decision matrix needs to be adjusted to meet the consistency requirement.
Assume that the evaluation condition set is as follows:where represents 5 score levels. The corresponding score is expressed in vectors. The corresponding evaluation standard may be expressed as
Determine the fuzzy matrix of . Calculate the value of . belongs to the dimension value marked . The member function is as follows:
Define the fuzzy evaluation vector of as :
The fuzzy compliance evaluation vector of is expressed as and may be obtained by using the following formula:wherewhere . Establish a fuzzy evaluation model and the evaluation value can be calculated using the following formula:
4. Personalized Exercise Recommendation Method Based on Knowledge Tracing
Aiming at the disadvantages of using collaborative filtering method and knowledge modeling method to generate personalized exercise recommendation, a personalized exercise recommendation method combining Deep Knowledge Tracing (KT) model and collaborative filtering method is proposed. KT uses Deep Knowledge Tracing model to model students’ learning status (i.e., learned knowledge) and then combines similar student information to perform collaborative filtering recommendation. The recommendation result considers both students’ learning status and the generality of group students’ learning status, thus improving the explanatory and accurate results.
4.1. Knowledge Tracing
Assume a time sequence of students doing exercises where a contact sequence of each student is input into a Recurrent Neural Network (RNN). The input sequence is represented by monothermal encoding. For example, if the input data involves exercises, each question is represented by 0 and 1, respectively, and the corresponding input length is 2. If the student answers the -th question incorrectly, the value of the -th position is 1 and those of the other positions are 0. Otherwise, the value of the -th + 1 bit position is 1 and those of the other positions are 0. After data is encoded and input into the deep neural network model, the output sequence of the model is established on the training of the exercise sequence of the students at the first moment. The length of the sequence is , and each bit position represents the correct answer probability of each corresponding question. By training the deep neural network model, the knowledge level vector of each student is calculated and the knowledge level vector matrix is obtained.
4.2. Score Prediction
For the knowledge horizontal vector matrix , the similarity between the vector and is determined by the cosine similarity. Assume that and , and the cosine similarity between and is calculated as follows:
A student whose cosine similarity is greater than a certain threshold (e.g., 0.9) is selected as a similar user by ranking the cosine similarity. For target student , a final exercise score vector of the target student is predicted according to the following formula:where is the student’s knowledge level vector, and is the average of similar students’ knowledge level vectors. The ratio of individual students’ knowledge level to the common knowledge level among students is adjusted by the parameter . The value range of is . As the value of increases, the impact on the student’s personal knowledge level increases. Particularly, when , is the average of the student’s knowledge level excluding the student’s personal status. When , is the level vector of personal knowledge output by the DKT model [20].
4.3. Recommendation
In this step, a recommendation result is generated for students. When creating an exercise recommendation, determine an appropriate difficulty range as the difficulty range of the problem, and then score the problem vector , and recommend the students with the correct probability of to . For example, setting to 0.6 and to 0. 8 will recommend questions with a correct answer probability of 0.6 to 0.8 to the student based on the predicted problem score vector. The proposed KT method takes into account the individual knowledge level of students and combines the similar group of students to the practice recommendation. The recommended practice can reflect the real knowledge level of students and recommend personalized exercises with certain difficulty to students. The Algorithm 1 is described as follows.
|
5. Personalized Opinion Model
Students’ comments on the course can reflect users’ preference for the online course to a certain extent. Therefore, this section uses the Text Classification (TC) model to score the comments of students and courses and uses the classic recommendation algorithm to recommend personalized course advice to teachers.
5.1. Scoring Model
The result of user comments on the course has an important impact on personalized recommendation. Course comments are short and have few valid information. The traditional text classification model does not have good effects [21]. In this paper, we use deep learning algorithm to score comments and use Word2Vector [22] to represent short text information as low-dimensional vectors. Based on this, we use neural network model to train comments. Compared with traditional methods, this method can improve the accuracy and recall rate of scoring models effectively. In this paper, the rating is −1/1. The value −1 indicates that you do not like the course (0 indicates that you do not like the course), and the value 1 indicates that someone likes the course. The definition of the problem is as follows: indicates the course review dataset. The feature set of each course review sample is represented by , and the corresponding category label is represented by : positive comment and negative comment. indicates the probability of positive and negative comments: . indicates the characteristic matrix of , indicates the total number of samples, indicates the total number of features , and indicates the prediction result of the -th sample.where indicates the threshold of the classification model. Usually, the value was 0.45 during the experiment.
On the basis of the problem definition, we design a specific neural network model to score the course review. Figure 3 shows the deep learning model of course review scoring.
The neural network consists of five modules. The input layer, convolution layer, pooling layer, fully connected layer, and output layer are in sequence. The input layer is used to cut the comment text and uses one-hot to represent the text. Convolution layer converts the one-hot feature of the input layer into a low-dimensional vector representation by using an embedding method and extracts text features by using different convolutions [23]. The pooling layer normalizes data at the convolution layer and converts convolution kernels of different dimensions into the same dimension. Fully connected layer learns linear relationships in the output of the pooling layer and learns more text features. The output layer is mainly used to predict samples. First, the ReLU function is used to perform nonlinear transformation on the linear relationship of the fully connected layer, and then the Softmax function is used to perform classification and scoring. The network structure can effectively learn the potential semantic space of short text and thus learn the student’s scoring prediction of the course.
5.2. Advice Recommendation Model
Based on the course score, the preference matrix of users is constructed by collecting the relationships among students, courses, and comments. Based on the matrix, the personalized learning model of online courses is trained by using the recommendation algorithm. The collaborative filtering recommendation algorithm is used for model training and prediction. A specific algorithm process is shown in Figure 4.
In the collaborative filtering recommendation algorithm, a preference matrix of is used to indicate the preference of a user for a course. A higher score indicates that the user likes the course. The value 0 indicates that the course is not selected. In Figure 4, the row of the preference matrix indicates support, the column indicates a course, and indicates a preference for course using . The collaborative filtering algorithm consists of two processes: prediction and recommendation. The prediction process is to predict the possible scores of users who have not selected courses. The recommendation process is based on the results of the prediction phase and recommends top N questions that students may have advice about.
6. Performance Evaluation
6.1. Settings
We leverage Keras [24] as basic neural model and use Python 3.7 library as coding tools. The FMOC algorithm is run on a workstation equipped with two GeForce GTX 1080Ti GPUs.
In order to verify the implementation effect of the proposed FMOC model in the online course learning process evaluation, this paper extracts and analyzes the background data of several major online course websites. In order to enhance the reliability of the verification results, the results of the formative evaluation optimization matching course were compared. The test data selected in this paper are extracted from website script data. Incomplete records are removed, and data information is classified by keyword index.
6.2. Results
On the whole, from the aspect of course matching optimization, this paper performs a comparison with the expert manual recommendation classification and verifies the reliability of the matching theory from another dimension. This test compares the automatic matching results of the proposed solution, the current course matching status on the website, and the classification results of experts’ manual suggestions, as shown in Figure 5.
According to the analysis in Figure 5, the matching degree of the proposed solution and expert suggestions is better than the result of no matching optimization on the website. It can also be seen that the judgment results of some courses in the proposed scheme are lower than those of experts, mainly because the lack of complete information guidance in the matching considerations leads to some deviations.
To evaluate the effect of personalized exercise recommendation, two sets of datasets are used. The first set is the ASSISTments [25] public dataset. ASSISTments is an open online education platform that can simultaneously teach and evaluate students’ learning; the dataset contains more than 500,000 records of submitted math questions. The second dataset is the history records of student submissions crawled from the online judge of a university. Table 1 lists the information about the two datasets.
In order to evaluate the effectiveness of the DKT-CF [20] method in generating personalized exercise recommendations for students, this paper firstly uses the indexes of precision, recall, and f1 to evaluate the recommendation effect; the formula for calculating the evaluation indicators is as follows:
In the DKT model’s training phase, 200 LSTM [26] nodes are used at the hidden layer, keep_prob is set to 0.5, learning_rate is set to 0.01, max_epoch is set to 200, and set batch_size is set to 32. The experimental results are shown in Tables 2 and 3. The following information can be obtained from the results: (1) In the experiments on the two datasets, the proposed DKT-CF algorithm is better than other algorithms. The accuracy, recall rate, and f1 value are all the highest. This indicates that the DKT-CF method can produce more accurate recommendation results compared to other methods and is more effective than other methods. (2) In the experiment, the accuracy of the UserCF algorithm using the traditional collaborative filtering algorithm is lower. After the students’ knowledge status is integrated, better results (KS-CF method) were obtained. The accuracy increased by 7.5% and 6.7%, respectively, and the method of introducing knowledge state information to improve the accuracy of traditional collaborative filtering is effective. (3) DKT-CF is a combination of DKT and KS-CF, which achieves the best effect; the method of considering both students’ personality and students’ study group in exercise recommendation can give the most effective personalized exercise recommendation.
Finally, to evaluate the effect of the personalized opinion model, an effect obtained by using the traditional word feature + SVM and deep learning improved algorithms in the training dataset is shown in Table 4.
As shown in the preceding table, the classification effect of the traditional word feature + SVM algorithm is slightly worse than that of the deep learning method proposed in this paper. We enrich the semantic information of short text by using Word2Vec word vector expression and then use the deep network structure to learn the semantic information. This function greatly improves the accuracy and recall rate of the scoring model, effectively identifies improvement suggestions, and provides basis for teachers to optimize courses.
7. Conclusions
Based on the analysis of three kinds of evaluation characteristics, this paper researches the formation evaluation into problem modeling. Through collecting the information of each link in the teaching process, comprehensive analysis, and evaluating the evaluation of online course teaching, we provide an optimization scheme for students’ online learning. Aiming at the fact that traditional collaborative filtering recommendation method does not consider students’ knowledge mastering level and the recommendation method based on knowledge modeling ignores the disadvantages of group learning among students, this paper proposes a personalized exercise recommendation method based on Deep Knowledge Tracing DKT-CF. Finally, the experiment proves the validity and rationality of the proposed DKT-CF method, which can improve the learning experience and efficiency of users.
In future research, we can improve the Deep Knowledge Tracing model and add more personalized features for learning. Based on the recommendation method, you can use other recommendation algorithms to generate recommendations. Based on recommendation scenarios, recommendation policies in different scenarios are also worth discussing.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest.
Acknowledgments
This work was supported by the Approved Project of Jilin Undergraduate Higher Education and Teaching Reform 2020 (General Project).