Abstract
This paper mainly studies the influence of dynamic cognitive diagnosis on personalized learning. Considering the influence of knowledge correlation factors and human brain memory factors on learning activities, a knowledge tracing model integrating learning factors is proposed. Firstly, based on the exercise-knowledge association information, the model maps learners and exercises to the knowledge space with clear meaning. Then, the evolution process of learners’ knowledge learning is quantitatively modeled in the knowledge space by integrating the classical learning curve and forgetting curve theory of pedagogy. On the other hand, considering the influence of topic semantics in the learning process, a knowledge tracing model integrating topic semantics is proposed in this paper. Firstly, the model designs a dynamic enhanced memory network to store the common information of knowledge and describes the learners’ dynamic mastery of knowledge. Secondly, the depth representation method of exercise resources is proposed to mine the text personality information and integrate it into the process of learners’ knowledge change modeling. Through a large number of experiments on exercise records, it is verified that the proposed model has accurate prediction performance and knowledge tracing interpretability.
1. Introduction
Personalized learning refers to an educational method in which appropriate learning resources and learning methods should be selected according to learners’ cognitive level, learning ability, and their own quality, starting from learners’ personality differences, so as to make up for the shortcomings of existing knowledge structure and obtain the best development. In traditional classroom education, students’ learning activities are completely formulated by teachers, limited by the constraints of time and space, and the exercise of personalized learning can only rely on teachers’ teaching experience. Therefore, in such an educational learning mode, it is difficult to formulate personalized learning programs for each student. With the deepening of educational informatization, a variety of online learning platforms have emerged. On the one hand, these online learning systems break through the time and space constraints in traditional classroom teaching. On the other hand, learners can choose learning contents and learning methods according to their own learning progress.
In the actual environment, learning activity is a long-term behaviour, and learners’ learning of knowledge is also a step-by-step process. Therefore, how to dynamically capture the changes of learners’ knowledge mastery is of great significance for the modeling of learners’ learning behaviour. Knowledge tracking task aims to track the changes of learners’ knowledge level through learners’ historical learning data. After years of research, knowledge tracking task has become one of the important research problems of online learning platform.
The research objects of this paper are learners, learning resources, and personalized learning mechanism. Learners are students who participate in learning activities, learning resources are exercise resources that have been recorded by various products and systems and can be provided to learner for learning, and personalized learning mechanisms are recommended strategies that can support and ensure learners’ personalized learning. How to understand learners’ learning process from an individual perspective and accurately describe and evaluate learners’ knowledge state and ability level is the core work and contribution of this paper.
Modeling learners’ learning activities and then evaluating learners’ learning status from different levels, such as cognitive level, skill level, and learning style, is the core task of educational data mining. In the early research and application, researchers proposed traditional measurement theories, such as classical test theory [1], generalization theory [2], and item response theory [3]. The new generation of measurement theory [4–6] represented by cognitive diagnosis aims to measure the knowledge processing process within learners, so as to analyze the knowledge state and skill level of students at the microlevel. Considering the dynamic characteristics of learners’ learning activities, the researchers further proposed the knowledge tracing model [7,8] in order to describe the dynamic change process of learners’ knowledge ability. The model can trace the changes of learners’ knowledge level through learners’ historical learning data. The early knowledge tracing research is represented by Bayesian knowledge tracing (BKT) model [9], which has strong assumptions that learners’ knowledge learning process would not be forgotten. In order to overcome the deficiency, Slater and Baker [10] divided the learning parameters of BKT model into two parts, namely, the learning part and the forgetting part. A large number of studies had improved the modeling ability of BKT model by adding external learning factors, such as difficulty factors [11], learners’ personality factors [12], knowledge hierarchy [13], and so on.
The traditional BKT model can only model learners’ learning on a single knowledge. Therefore, it usually regards learners’ learning process of multiple knowledge as multiple discrete processes. In order to jointly trace learners’ learning status on multiple knowledge, Yeung et al. [14] used the cyclic neural network model to model learners’ learning records and proposed a deep knowledge tracing (DKT) model, which breaks the independence between knowledge, models the comprehensive state of learners’ multiple knowledge points in hidden space, and has stronger semantic expression ability. With the performance of neural network, DKT has attracted the attention of a large number of researchers. On the basis of DKT, Yang and Cheung [15] used decision tree to fuse a variety of exercise attribute features. Su et al. [16] fully considered the process of memory and forgetting in learners’ learning and proposed three forgetting characteristics of temporal correlation, which further improved the temporal dependence of DKT model. Rozewski et al. [17] considered the dependence at the knowledge level, proposed the dynamic key-value memory networks, and used the memory network to model the influence process of knowledge. Liu et al. [18] combined the sequence relationship at the knowledge level, which constrained the performance of DKT model at the prediction level. Sun et al. [19] used graph neural network to model propagation process of learners’ knowledge mastery in the learning process.
2. Knowledge Tracing Model Integrating Learning Factors
The core task of personalized learning is knowledge level diagnosis, that is, to evaluate learners’ mastery of different knowledge concepts. The task of knowledge level diagnosis is to evaluate the changes of learners’ mastery of various knowledge concepts based on the learning process. In order to describe the change of learners’ learning ability level, in the field of educational psychology, learners’ learning state is characterized as a comprehensive ability value or a set of binary vectors of knowledge mastery. From the perspective of data mining, learning from the score prediction task, the matrix decomposition model projects learners in the hidden space to describe their knowledge level. Although these two methods have achieved some results, they both ignore the important learning factors in learners’ learning process.
Existing studies show that learners’ learning process is usually affected by learning factors from two levels: knowledge and learners. Although the existing work preliminarily discusses the relevance of knowledge level, there is a lack of direct mining of the topic relevance reflected from the knowledge level. From the learner level, the research of educational psychology shows that its learning process is a dynamic and complex process. The brain’s memory and forgetting of knowledge make the learners’ knowledge level change continuously. Two types of pedagogical studies illustrate this phenomenon. Learning curve theory [20] shows that, with continuous attempts and practice, learners can acquire relevant knowledge. Forgetting curve theory [21] shows that, with the passage of time, learners’ memory of knowledge is getting worse and worse, which reflects the downward trend of their knowledge level. The knowledge tracing model constructed in this paper takes the learning and forgetting factors at the learner level as additional parameters, so as to better trace and explain the reasons for the changes of learners’ knowledge level.
In this paper a knowledge tracing model integrating learning factors is proposed, which dynamically traces and explains the changes of learners’ knowledge level from the perspective of probability modeling. Combined with the topic knowledge incidence matrix, the model first maps the topic to the knowledge space. In the knowledge space, each topic is represented by a knowledge vector, and each dimension of the vector represents a clear knowledge concept. Considering the correlation factors at the knowledge level, the model finds a neighbor topic set with the same knowledge concept for each topic and then gathers the knowledge vector information of neighbor topics for each topic, so that the topics with knowledge correlation characteristics have a similar distance in space. Secondly, according to learners’ learning performance, the model projects learners into the same knowledge space at each time. Each learner is represented by a horizontal vector, and each dimension of the vector represents his mastery of the corresponding knowledge concept at that time. Then, the model combines learning curve theory and forgetting curve theory to quantify the learning factors of learners’ memory and forgetting in the process of dynamic learning, so as to capture the changes of their knowledge.
2.1. Knowledge Level Diagnosis Problem Definition
The online learning system includes learners, topics, and knowledge concepts. The learning log data can be expressed as a tensor .
The learners’ learning log can be recorded as shown in Table 1.
The corresponding topic knowledge incidence matrix is shown in Table 2.
The knowledge level diagnosis problem studied in this paper can be formally defined as follows: input the learning log tensor and the corresponding knowledge-topic incidence matrix , the task objectives of the model include: (1) At every moment, evaluate learners’ mastery of knowledge and concepts, and track the change of their mastery level. (2) Predict learners’ mastery level of knowledge concepts at moment, and predict their scores for answering questions.
The mathematical symbols in the model definition are shown in Table 3.
Similar to the common probability model, for each learner and topic , the model combines the learner level vector and topic knowledge vector to model the conditional probability distribution of learning log tensor as follows:where represents normal distribution with mean and variance , represents log tensor, if learner studies topic at time, and then . represents inner product of two vectors.
In order to trace the changes of learners’ knowledge level, it is assumed that learners’ level of knowledge concepts changes over time. Therefore, the knowledge level of all learners can be expressed as a tensor composed of a set of matrices in the case of time series: .
Then, the topic knowledge matrix integrating knowledge partial order factors and knowledge correlation factors is constructed. Because each dimension of learner tensor and topic knowledge matrix cannot be related to any clear knowledge concept, it is necessary to project the topic into a knowledge space with clear meaning by combining the topic knowledge incidence matrix . However, because matrix is usually marked manually, it has subjective error. In order to solve this problem, based on the partial order learning scheme, this paper improves the definition of the traditional matrix, so as to improve the effect of probability modeling under the requirement of ensuring the interpretability of knowledge space.
Specifically, for a topic , its partial order relationship about all knowledge concepts is defined firstly as follows:
Based on the partial order relation, the ternary comparable set of matrix can be constructed as follows:
Each topic contains some knowledge concepts, which shows that topics with the same knowledge concepts have similar characteristics in their knowledge space. Intuitively, learners’ scores on such topics are usually consistent, which can help the knowledge level diagnosis task. Specifically, for topic , its neighbor set with the same knowledge concept is defined firstly:
Therefore, the model assumes that the knowledge vector information of the topic is jointly affected by its neighbor topic :where represents influence weight of neighbor topic on topic .
2.2. Pedagogical Application of Model
The results of the knowledge level diagnosis task can be applied to multiple scenarios of educational analysis.
Knowledge ability prediction [22]: the goal of knowledge ability prediction is to predict the probability that learners will master a certain knowledge concept in the future. The prediction results could help learners master their learning status in time and select appropriate learning resources, so as to avoid wasting time learning the knowledge they have mastered.
After model training, the knowledge level matrix of learner at the current time and his personalized learning parameter can be obtained. Therefore, through calculating his memory factor and forgetting factor at time , his knowledge ability level for each knowledge concept at time can be predicted.where represents prediction ability of learner on all knowledge concepts at time , represents the interval between time and its closest learning time about knowledge concept , and represents frequency and times of learners practicing knowledge at time.
Grade prediction: the goal of grade prediction is to predict the score of learners on the questions they have not done. The prediction results usually reflect learners’ familiarity with relevant knowledge and concepts. According to the learners’ knowledge level, we can predict the learners’ scores for the exercises that have not been done.
For learner and topic , after model training, the knowledge vector and difficulty coefficient of topic can be obtained directly. Then, the knowledge level vector of learner at can be predicted. The score of learner on topic can be predicted as follows:
3. Knowledge Tracing Model Integrating Topic Semantics
This model focuses on the influence of knowledge commonality information and personality information on learners’ learning behaviour process, so as to solve the problem of performance prediction. Specifically, when modeling, input the topic sequence made by learners in history. Firstly, in the model a dynamic enhanced memory network to store the common information of each knowledge concept is proposed, and the influence of topic knowledge on learners’ level of different knowledge concepts at each time is quantified. Secondly, the model proposes a topic personality feature extractor, which uses two-way recurrent neural network [24] to understand the meaning of the topic and mine the personality characteristics of the topic content. Then, integrating the influence of common knowledge and individual content characteristics at each time, a multidimensional LSTM network is proposed to model and track the changes of learners’ multiple knowledge concept states at the same time.
3.1. Semantic-Based Topic Representation Prediction
The proposed model is oriented to language exercises; taking English reading comprehension as an example, a prediction model integrating semantic representation is designed to improve the prediction accuracy and stability.
The core of the proposed method is to understand the semantic meaning of exercise and then analyze its difficulty level from the text level. The proposed method is a two-stage framework, including training stage and testing stage. Firstly, in the training stage, combined with the historical test data and the corresponding exercise text, comprehensively understand multiple content parts of the text in each topic , so as to represent its semantic information and predict its difficulty . In the test stage, the representation method of topic based on semantic understanding can directly input the text of the topics to be practiced and directly predict their difficulty attributes.
The representation method of topic based on semantic understanding consists of four layers: input layer, sentence understanding layer, semantic association layer, and prediction layer. Each part of the text contained in the topic input by the input layer initializes the corresponding representation according to the word. The sentence understanding layer uses the unified CNN model to segment the topic text into sentences for semantic understanding from the sentence level. The semantic association layer uses the attention mechanism to quantify the semantic dependence of each topic on different sentences in the exercise and obtain the semantic representation results of the topic. The prediction layer splices the semantic representation of each part of the topic and the prediction difficulty.
The input of the method is the text content of each part of a topic , including background document , topic , and option . The document consists of a sequence of statements , where represents the i-th sentence and represents the number of sentences in the document. Topic and each option are modeled as a separate sentence. Each sentence consists of a sequence of words , where represents the number of words in each sentence, each word is represented by a pretrained word embedding vector with dimension , and .
The input of the sentence understanding layer is each sentence matrix, . In this paper, we use the wide convolution operation (the convolution kernel size is ) to understand the local semantics of every word at the sentence level. After the first layer convolution operation, the input sentence can obtain the hidden layer sequence , where each hidden layer element is defined as follows:where , is parameter of convolution operation, is dimension of output vector, is nonlinear activation function, and is splicing operation.
In the above convolution operation, the model can directly learn the local semantic information in each word in the sentence. The model alternately uses multilayer similar convolution pooling operations to gradually learn the global semantic information at the statement level and finally obtain a sentence embedded vector , where represents the dimension of the sentence embedded vector.
After the sentence understanding layer obtains each sentence embedded vector at the text level, the semantic association layer learns the semantic representation of the text from the reading topic level. Specifically, the topic synthesis representation would aggregate the weighted sentence semantic information from the document level and the option level, respectively. For each topic , the attention vector at the document level is defined as follows:where represents the j-th sentence of document , is cosine similarity, it quantifies the dependence of each sentence in document on topic , and it can be defined as follows:where represents sentence representation vector of topic.
enhances the interpretability of the model; the higher the weight score, the higher the dependence of topic on the corresponding sentence.
The prediction layer uses the fully connected network to learn the comprehensive semantic representation of each topic and then predicts the corresponding difficulty .where , , , and are network model parameters.
3.2. Knowledge Tracing Model Integrating Topic Semantics Based on Attention Mechanism
In the process of modeling, the input of the model is the learner’s history exercise sequence. Suppose that the learner’s knowledge level is a matrix containing knowledge level vectors to represent the learner’s mastery of all knowledge concepts.
Learners’ knowledge level at each time can be defined as follows:where represents dimension of knowledge level vector.
At time , the goal of the topic content module is to understand the semantic representation of the topic made by the current learners, so as to reflect the personality characteristics of the topic and distinguish the characteristics of different topics. The topic content module represents the semantics of the topic text content from positive and reverse aspects:where represents positive semantic representation of word , represents negative semantic representation of word , and splices the semantics of word in two directions, so as to ensure its semantic information to the greatest extent.
After obtaining the semantic representation of words, the semantic representation of topic can be obtained by aggregating all words through the maximum pooling operation at the element level.
The goal of the learning process module is to model the learner learning sequence, so as to track the state change of the knowledge level matrix . The model uses recurrent neural network to realize the learning process module. At time , in order to distinguish the different effects of learners’ correct () and wrong () answers to a topic on their knowledge state, a merging method is designed to aggregate the topic semantics and learners’ answer results. Then, the state of each knowledge level of learners is updated and modeled.
For the knowledge concept , the learner’s knowledge state of can be jointly affected by the aggregation vector at the current time and the knowledge state at time.
In fact, RNN can be implemented in many forms. Considering the long-term dependence of learners’ learning sequence, in this paper LSTM model is used.
Through the modeling process, the model can obtain the learners’ knowledge level from time 1 to . The research of educational psychology shows that learners’ performance on the topic usually depends on learners’ historical learning status and the semantics of the current topic. Therefore, this paper proposes a prediction strategy based on attention mechanism. The prediction process requires two parts of information: learners’ historical knowledge and corresponding topic representation .
The model assumes that the knowledge state of learners at time is , which is the weighted aggregation result of their historical time state. Its weight is reflected by the similarity between time topic and historical topic .
At time , the attention vector of the i-th concept in knowledge state is defined aswhere represents semantic representation of topic , cosine similarity represents the attention weight, and it measures the contribution of topic to .
4. Experiment and Analysis
This section makes experimental analysis on the knowledge tracking model integrating learning factors (KTLF) and knowledge tracking model integrating topic semantics (KTTS).
Two experimental datasets were used in the experiment. ZHS dataset is the independent exercise data collected from an online education platform (https://www.zhihuishu.com); Assist is a public dataset of students’ online mathematics learning records. The statistical information of datasets is shown in Table 4.
4.1. Effect Evaluation of Knowledge Tracking Model Integrating Learning Factors
First, the parameters in the learning factor and forgetting factor are set, , . The benchmark methods item response theory (IRT) [25] and learning factor analysis model (LFA) [26] in educational psychology and probability matrix factorization (PMF) [27] and deep knowledge tracking model (DKT) [15] in data mining are selected, and the knowledge tracking models QMIRT [28] and QPMF [29] are selected to compare with KTLF model.
In this experiment, mean absolute error (MAE) and root mean square error (RMSE) were used as evaluation indexes. Figures 1 and 2 show the experimental results of learner performance prediction.
From Figures 1 and 2, it can be seen that KTLF model performed best in both data pieces. It shows that the introduction of learning factors is necessary for the task of knowledge level diagnosis. Then, the effects of QMIRT and QPMF are better than the traditional IRT and PMF models, which once again shows that the modeling with knowledge factors is effective. To sum up, the modeling process of knowledge tracking model integrating learning factors is effective and necessary.
4.2. Effect Evaluation of Knowledge Tracking Model Integrating Topic Semantics
The benchmark methods item response theory (IRT) and learning factor analysis model (LFA) in educational psychology and probability matrix factorization (PMF) and deep knowledge tracking model (DKT) in data mining are selected, and the knowledge tracking model LSTMM [30] is selected to compare with KTTS model.
Four evaluation indexes were selected from the perspectives of regression and classification. From the perspective of regression task, the experiment uses absolute mean error (MAE) and root mean square error (RMSE) to measure the error between the predicted value and the real value. The smaller the index value, the better the prediction result. From the perspective of classification task, the experiment regards learners’ correct answers as positive examples and learners’ wrong answers as negative examples. Accuracy (ACC) and ROC curve area (AUC) were used. Generally, when ACC and AUC are 0.5, it indicates that the prediction result is random. The larger the index value, the better the prediction result of the model.
The experiment compared and analyzed all the models in the learner performance prediction task. Firstly, the dataset is divided from the learner level, and the learning sequence of each learner is divided into training set and test set. For a learner, the experiment divides the first 60%, 70%, and 80% of the records in the topic sequence into training sets, and the rest into test sets. The experiment was run for 5 times, and the average value was taken as the final result. Figures 3–6 show the experimental results of all models.
From Figures 3 and 6, it can be seen that KTTS model performed best. It shows that the proposed method in this paper can integrate the topic semantics, so as to improve the modeling effect of students’ learning records and improve the prediction accuracy. Then, the prediction effect of the model based on attention mechanism (KTTS) is better than that based on Markov property (LSTMM), which shows that learners’ performance on future topics is directly related to their historical records, and learners’ performance on similar topics is relatively consistent. Therefore, dynamically capturing important states in history through attention mechanism has an important positive impact on prediction tasks. In addition, the model containing the topic content module can accurately model the topic text, so as to better mine the personality information of the topic and distinguish the influence of different topics on learners’ learning.
5. Conclusions
This paper provides a set of data-driven personalized learning tracking models based on learners’ learning activity data. Aiming at the semantic and structural complexity of exercises, the depth representation method and application of exercises are carried out. English reading comprehension is chosen as a typical language exercise, and the in-depth representation method and application of exercise resources are studied from two aspects: semantic understanding and structural understanding. From the perspective of students, this paper studies the dynamic cognitive diagnosis method in their learning process. Firstly, considering two kinds of learning factors, namely, knowledge correlation factor and human brain memory forgetting factor, this paper proposes a knowledge tracking model integrating learning factors. On the other hand, considering the influence of exercise topic semantics on learning behaviour at the level of knowledge commonness and text individuality, this paper proposes a knowledge tracking model integrating topic semantics. The experimental results show that the proposed models in this paper can capture the important state of learners’ learning process from the topic semantic level and has good results in historical knowledge tracking and future performance prediction. Although some exploratory work has been done in this paper, there are still many important scientific problems and application directions worthy of further discussion and research. Firstly, it is greatly significant to study the understanding and representation technology of multimodal learning resources. Secondly, for students, the study of interpretable cognitive diagnosis is the core issue in the future.
Data Availability
The basic data used in this paper are downloaded from https://github.com/topics/assistments.
Conflicts of Interest
The author declares that there are no conflicts of interest regarding the publication of this study.
Acknowledgments
This work was supported by a grant from Humanities and Social Science Project of Shandong Province (no. 2020-ZXGJ-29) and a grant from Social Science Planning Research Project of Shandong Province (no. 19CPYJ101).