Abstract
Due to the shortcomings in the traditional methods which dissatisfy the examination requirements in composing test sheet, a new method based on tabu search (TS) and biogeography-based optimization (BBO) is proposed. Firstly, according to the requirements of the test-sheet composition such as the total score, test time, chapter score, knowledge point score, question type score, cognitive level score, difficulty degree, and discrimination degree, a multi constrained multiobjective model of test-sheet composition is constructed. Secondly, analytic hierarchy process (AHP) is used to work out the weights of all the test objectives, and then the multiobjective model is turned into the single objective model by the linear weighted sum. Finally, an improved biogeography-based optimization—TS/BBO is proposed to solve test-sheet composition problem. To prove the performance of TS/BBO, TS/BBO is compared with BBO and other population-based optimization methods such as ACO, DE, ES, GA, PBIL, PSO, and SGA. The experiment illustrates that the proposed approach can effectively improve composition speed and success rate.
1. Introduction
Examination is the process of teaching management which plays an important role in evaluating student achievement, inspiring students’ creativity, and improving student learning outcomes. As an important means of testing student learning and teacher teaching, it also provides the necessary information for teachers to improve their teaching methods and the quality of teaching [1]. Also, tests are often used to help instructors assess students’ prior knowledge [2]. With the advent of computer based education (CBE), how to take full advantage of computers to replace manual to help the examination organizer deal with a variety of complex repetitive work is an important topic in computer managed instruction, so computer-aided test (CAT) come into being. Computer-aided test can best satisfy the different evaluation requirements; quickly and effectively complete test-sheet composition with randomness, scientificity and rationality.
Test-sheet composition is a hot issue in computer-aided testing. At present, the algorithms to solve test-sheet composition are abductive machine learning [3], harmony search algorithm [4], genetic algorithm [5, 6], and particle swarm optimization [7]. However, these algorithms has many shortcomings, such as, meeting the local constraints, parameter uncertainty, the lack of random selection and too long time required for composing test-sheet, which lead to dissatisfy evaluation requirements.
Firstly proposed by Simon in 2008, biogeography-based optimization (BBO) [8–10] is a novel powerful metaheuristic algorithm, which has good global optimization capability, insensitivity for selection of initial values and parameters, strong robustness, simplicity, and easy implementation. Therefore, it has been widespread concern in the academic research and application and penetrated to a number of fields [11]. However, in the field of test-sheet composition, no application of BBO algorithm exists yet. In this paper, we use an improved BBO algorithm to solve the test-sheet composition problem. To reduce the search space and improve efficiency, an improved BBO algorithm makes use of subencoding based on question type. Moreover, we use tabu search (TS) to avoid duplication of encoding in the migration and mutation operations. Analytic hierarchy process (AHP) is used to determine the weight of the test-sheet goal, and then multiobjective optimization model is converted to a single-objective model by a linear weighted sum. In [12], Hwang et al. use tabu search to solve a set of parallel test sheets with identical test ability and get a good performance. Here, we combine BBO and TS to construct a new algorithm according to AHP and the principle and feature of TS and BBO; that is, we use TS to optimize the mutation and migration operator in BBO and then design the TS/BBO algorithm to get the most optimal question combination to form a testsheet.
The structure of this paper is organized as follows. Section 2 describes the mathematical model in test-sheet composition problem. In Section 3, the weight in test-sheet composition is determined by AHP. Subsequently, preliminary knowledge of TS and BBO algorithm are introduced in Sections 4 and 5, respectively. Then, an improved BBO algorithm for test-sheet composition is presented in Section 6 and the detailed implementation procedure is also described. The simulation experiments are conducted in Section 7. Finally, Section 8 concludes the paper and discusses the future path of our work.
2. Problem Description
Test-sheet composition model is a multiconstraint, multiobjective optimization problem. In this section, we will describe the mathematical model for test-sheet composition, including constraints and objective function.
2.1. Constraints
Automatic test-sheet composition with computer is searching for questions to meet the requirements from item bank. We determine the encoding for the main properties according to objective analyzing and preestablish the goal state matrix A for all the questions in the item bank: where M is the total number of items in the item bank; is the number of attributes for each item (here, ). For goal state matrix A, each row represents all properties every item, and each column represents a property of the entire item in the item bank. It has not more than constraints because () means item number.
The process of composing testsheet is as follows. Firstly, find the row to meet the objectives and requirements of question combination in the state matrix A according to the default settings or parameters user input, where is a vector composed by the question type is the number for question in the type , where is the type number of questions in the item bank, is the number of the th question, and , is the total question number in a testsheet. For simplicity, we use Boolean vector instead of to represent the required question combination at last. Here, is the state variable for the mth row (i.e., the mth item or question), where, means the mth row selected, that is, there exists ; means the mth row not selected, that is, there does not exist . To simplify, without loss of generality, we use to represent the question selected or not.
Every objective for test-sheet composition corresponds to a constraint, all the test-sheet objectives corresponding to the following (eight) constraints. Thus, the mathematical model, that is, the constraints for goal state matrix A, can be constructed as follows:(1) Total score where is total score for the test-sheet composed; is score for the question.(2) Test time where is the total time completing the test-sheet composed; is estimated time for the completion of the question.(3) Chapter scores where is the score of the jth chapter; is the chapter number for the question.(4) Knowledge point score where is the score for the jth knowledge point; is the knowledge point number for the question.(5) Question type score where is the score for the jth question type; is the question type number for the question. All the questions in item bank are divided into four kinds (i.e., Type 1, Type 2, Type 3, and Type 4);(6) Cognitive level score where is the score for the jth cognitive level; is the cognitive level number for the question.(7) Difficulty degree where is the difficulty degree of the test-sheet composed; is difficult degree for the question.(8)Discrimination degree where is the discrimination degree of the test-sheet composed; is discrimination degree for the question.
2.2. Objective Function
Through analysis, eight constraints of the test-sheet composition model (total score, test time, chapter score, knowledge point score, question type score, cognitive level score, difficulty degree, and discrimination degree) should be equal to the evaluation requirements or with the minimum error of the evaluation requirements. Thus, the test-sheet composition is a multiconstraint, multiobjective optimization problem. In practical application, we can consider the error between the above eight constraints and evaluation requirements as objective function . Therefore, the composed paper is optimal when reaches minimum.
The mathematical model (i.e., objective function) for the test-sheet composition is as follows:
In the above objective function , , , , , and are the positive deviation between the test-sheet property and evaluation requirements, that is, the part beyond the evaluation requirements; , , , , and are the negative deviation between the test-sheet property and evaluation requirements, that is, the part less than the evaluation requirements. The positive deviation and negative deviation cannot exist at the same time; therefore, their product will be 0, that is, ; is weight for test-sheet composition, whose sum is 1.
In practice, the weight has an important impact on the test-sheet composition whether the result satisfies the requirements. Therefore, how to scientifically and reasonably determine the weight of the test-sheet goal is worthy of further study. In the next section, we will use AHP to determine the weights.
3. Determining the Weight in Test-Sheet Composition Using AHP
AHP [13] is an effective combination of qualitative and quantitative analysis of multiobjective evaluation method, proposed by the University of Pittsburgh professor Saaty in the 1970s. Advantages of AHP are easy implementation, simple structure, speed, and robustness. Due to these advantages, it has many real-world applications, such as evaluating course website quality, improving the service quality of e-learning, and evaluating the web-based e-learning system.
3.1. AHP Algorithm
Proposed by Zahedi [14], four main steps are identified in using the AHP to solve a decision-making problem. The detailed algorithm AHP is described in Algorithm 1.
|
3.2. Determining the Weights Using AHP
AHP can be used to determine goal weight for the test-sheet composition. Integrating the feature of test-sheet composition into the AHP, specific solution process is shown in Algorithm 2 (Figure 1).
|
We calculate consistency ratio for the compassion matrix with Algorithm 2 and get the weights as shown in Table 2, so we can draw a conclusion that the compassion matrix has satisfactory consistency.
In the actual exam, test-sheet goal weight is various. AHP can determine different test-sheet goal weight responding to different requirements according to the different compassion matrix C.
4. Biogeography-Based Optimization (BBO)
Biogeography is the study of the migration, speciation, mutation, and extinction of species [15]. Biogeography has frequently been thought of as a process that compels equilibrium in the number of species in islands. However, equilibrium in a system can also be viewed as a minimum-energy configuration, so we see that biogeography can be considered as an optimization process. This idea is further discussed in [16].
Firstly proposed by Simon in 2008, biogeography-based optimization (BBO) is a new evolution algorithm developed for the global optimization [8]. It is inspired by the immigration and emigration of species between islands (or habitats) in search for more compatible islands. Each solution is called a “habitat” (or “island”) with an HSI (habitat suitability index) and represented by an n-dimension real vector. An initial individual of the habitat vector is generated at random. The habitat with a high HSI is considered to be good solution, while the habitat with a low HSI is considered to be bad solution. Low HSI can take in many new good features form the high HSI, so that these low HSI solutions have a relatively high possibility that become high HSI solutions. In BBO, habitat H is a vector of n (suitable index vectors (SIVs)) initialized randomly and then implements migration and mutation operator to achieve the optimal solution. The new candidate solution is generated from the entire habitat in population by using the migration and mutation operators.
In BBO, migration operator can change existing habitat and modify existing solution. Migration is a probabilistic operator that adjusts habitat . The probability modified is proportional to its immigration rate , and the source of the modified probability from is proportional to the emigration rate . Migration operator is shown in Algorithm 3.
|
Mutation is also a probabilistic operator that randomly modifies habitat SIVs based on the habitat a priori probability of existence. Very high HSI solutions and very low HSI solutions are equally improbable. Medium HSI solutions are relatively probable. The mutation rate is expressed as where is a user-defined parameter.
Additionally, the mutation operator tends to increase the population diversity. Mutation can be described in Algorithm 4. The basic framework of BBO algorithm can be simply described in Algorithm 5. More details about the migration operator, mutation operator, and BBO algorithm can be found in [8] and in the Matlab code [17].
|
|
5. Tabu Search (TS)
Tabu search (TS) [18] is a metaheuristic framework which takes advantage of its use of adaptive memory strategies to manage simultaneously the variety and intensification searches in the solution space of the optimization problem under way. The TS framework is composed by the following major components.
Solution Configuration
The candidate solution for the optimization problem under way is encoded by a configuration. Relying on the characteristic of the problem, the solution configuration , which contains decision variables, could be a vector taking real numbers, binary values, or mixed combinations of them. Initially, a configuration is set randomly within the reasonable ranges of the decision variables.
Move Function
A move function is used to guide the solution configuration to explore its local solution space. The step size of the move should be limited in an appropriate range according to the practical problem. An oversized step makes the search to step over a promising neighbor, while an undersized step causes the algorithm to be inefficient. By performing the move function, the solution configuration is turned into a neighboring configuration, that is, , where is a neighboring configuration of .
Neighborhood
The move function defines a neighborhood which bounds the neighboring configurations which are reachable by implementing a move operation to the current solution configuration using . Generally, the neighborhood of solution is defined by .
Tabu List
A tabu list which stops a recent move to be reversed is kept and modified in the TS framework. Once a move (say, swapping the th and the jth entries of ) is carried out, the inverse of this move (swapping the two entries again) is recorded in the tabu list and is labeled tabu-active. All the tabu moves are excluded from the neighborhood , and thus become inaccessible.
Aspiration Criterion
To allow the solution configuration to move to an attractive but tabu neighboring configuration, an aspiration criterion is strategically designed to overrule the tabu status of such a desired move. Aspiration criterion provides a restricted degree of freedom in accepting a tabu move that achieves a threshold of attractiveness.
Stopping Criterion
The stopping criterion depends on the purpose of the problem. There are a number of alternatives such as a minimal solution quality level, a given CPU time limit, a maximal number of iterations between two improvements of the best solution found.
Basically, the TS approach can be summarized as shown in Algorithm 6.
|
6. Tabu Search/Biogeography-Based Optimization (TS/BBO)
As we all know, the standard TS algorithm is good at exploring the search space and locating the region of global minimum, but it is relatively slow at exploitation of the solution. On the other hand, standard BBO algorithm is usually quick at the exploitation of the solution, though its exploration ability is relatively poor. Therefore, in our work, a hybrid metaheuristic algorithm by integrating tabu search into biogeography-based optimization, so-called TS/BBO, is used to solve the problem of test-sheet composition. The difference between TS/BBO and BBO is that the hybrid migration operator is used to replace the original BBO mutation operator. In this way, this method can explore the new search space by the mutation of the TS algorithm and exploit the population information with the migration of BBO, and therefore it can overcome the lack of the exploitation of the TS algorithm. In the following, we will show the algorithm TS/BBO which is a variety of TS and BBO. Firstly, we describe tabu search migration and mutation operation, and then a mainframe of TS/BBO is shown.
6.1. Tabu Search Migration and Mutation Operation
The critical operator of TS/BBO is the tabu search migration operator, which composes the tabu search with the migration of BBO. In this algorithm, we can find that the migration between the population and has no repeat because of the use of tabu search. The core idea of the proposed hybrid migration operator is based on two considerations. First, poor solutions can receive many new features from good solutions. Second, the final solution cannot include any repeat SIV. Pseudocode of tabu search migration operation can be described in Algorithm 7.
|
In the same way, we compose the tabu search and original mutation operator which modifies the habitat with low HSI in order to avoid the repeat to improve the search efficiency. Pseudocode of tabu search mutation operation can be described in Algorithm 8.
|
6.2. Mainframe of TS/BBO
By incorporating above-mentioned tabu search migration and mutation operator into original BBO algorithm, the TS/BBO has been developed as a new algorithm. TS/BBO algorithm is given in Algorithm 9.
|
6.3. Algorithm TS/BBO for Test-Sheet Composition
In TS/BBO, the standard continuous encoding of TS/BBO cannot be used to solve test-sheet composition directly. In order to apply TS/BBO to test-sheet composition, one of the key issues is to construct a direct relationship between the test-sheet sequences and the vector of individuals in TS/BBO.
6.3.1. Preprocessing
We will do the following preprocessing before the design of TS/BBO algorithm for test-sheet composition(a)We regroup all the questions in item bank according to question type, and then the candidates take the same question type together to form a subset. So all the questions in item bank can be divided into several different subsets, and then all questions will be renumbered.(b)Because question type in the requirements and item bank has the same score, we can calculate the number of questions required for every question type, and then we get the total number of questions in a testsheet.
6.3.2. Encoding
When using TS/BBO to solve test-sheet composition problem, the status code for each habitat represents a candidate solution, that is, a test-sheet composition scheme. Therefore, how to determine the effective habitat status code is a key issue.
Traditional encoding method is as follows: the status code of each habitat is represented by a binary string whose length is the number of questions in total in item bank, and the number “1” indicates that the question corresponding to the number is selected, while number “0” indicates that the question corresponding to the number is not selected; the length of the number “1” indicates the number of questions contained in total in a testsheet. This encoding method is simple and maximizes a random search at most, but it has increased the search space, reducing the search efficiency.
Therefore, an alternative encoding method is proposed in this paper. This method rearranges questions into different subsets according to the different question type. The status code of each habitat is represented by an l-dimensional vector . The dimension for vector depends on total questions in a testsheet, where the vector is composed by the question type, the integer means the jth question in the question type; , is the total question in item bank; is the number of the question type, and , is the total question in some test-sheet composed.
For example, we have seven questions in item bank; questions a and g belong to question type 1, 1 score for each question; questions b, e, and f belong to question type 2, 2 scores for each question; questions c and d belong to question type 3, 3 scores for each question. Questions in item bank will be rearranged and renumbered according to question type and the result is . According to the question type requirement {question type 2 : question type 3} = {4 : 3}, we get a total of 3 questions in the testsheet, that is, . , , , , and . So (4, 5, 7) represents a test-sheet composition scheme; that is, it represents a test-sheet containing three questions which are questions e, f, and d.
6.3.3. The Algorithm TS/BBO for Test-Sheet Composition
Improved BBO can adapt to the needs of testsheet, while optimization algorithms can improve the BBO fast search capabilities and increase the search to the global possible optimum solution. HSI in Habitat is represented by the objective function in test-sheet composition model the smaller the value , then the higher HSI in Habitat .
Based on the above analysis, the pseudo code of improved BBO-TS/BBO for test-sheet composition is described as shown in Algorithm 10.
|
7. Simulation Experiments
In this section, we look at the performance of TS/BBO as compared with BBO and other population-based optimization methods. Firstly, we compare performances between BBO and TS/BBO, and then we compare performances between TS/BBO and other population-based optimization methods such as ACO, DE, ES, GA, PBIL, PSO, and SGA.
To allow a fair comparison of running times, all the experiments were performed on a PC with a Pentium IV processor running at 3.0 GHz, 1 GB of RAM and a hard drive of 160 Gbytes. Our implementation was compiled using MATLAB R2012a (7.14) running under Windows XP. In the following, we will describe the problem we use to test the performance of the TS/BBO.
7.1. TS/BBO versus BBO
We randomly generate an item bank with M (e.g., ) questions, and then choose some questions to form a testsheet. Its constraints are shown as follows:(1)the total score is 100;(2)the test time is 120 minutes;(3)chapter score {1 : 2 : 3 : 4 : 5 : 6 : 7 : 8 : 9 : 106 : 6 : 8 : 12 : 9 : 12 : 12 : 15 : 10: 10};(4)knowledge point score {1 : 2 : 3 : 4 : 5 : 6 : 7 : 8 : 9 : 10:11:12:13:14:15:16:17:18:19:20} = {3 : 3 : 3 : 3 : 4 : 4 : 6 : 6 : 4 : 5 : 6 : 6 : 6 : 6 : 6 : 9 : 5 : 5 : 5 : 5} (a chapter contains two knowledge points);(5)question type score {Type 1 : Type 2 : Type 3 : Type 4} = {10 : 16 : 24: 50};(6)cognitive level score = ;(7)difficulty degree is 0.4900;(8)discrimination degree is 0.4050.
We get the goal weights for test-sheet composition {total score : test time : chapter score : knowledge point score : questions type score : cognitive level score : difficulty degree : discrimination degree0.3423 : 0.1412 : 0.0792 : 0.0319 : 0.2244 : 0.0417 : 0.0970 : 0.0424} according to the Algorithm 2 as shown in Section 3.2.
For BBO and TS/BBO, we used the same following parameters: habitat modification probability = 1, immigration probability bounds per gene = , step size for numerical integration of probabilities = 1, maximum immigration and migration rates for each island = 1, and mutation probability = 0.005. We must point out that TS/BBO needs to set tabu search table except for the above parameters. To compare the different effects among the parameters Popsize, Maxgen, and Keep (the number of elitisms), we ran 100 Monte Carlo simulations of BBO and TS/BBO algorithm on the above problem to get representative performances. Tables 4, 5, 6, and 7 show the results of the simulations. In other words, Tables 4, 5, and 6 shows the best, worst, and average performance of BBO and TS/BBO algorithm found by BBO and TS/BBO algorithm over 100 Monte Carlo runs, respectively, while Table 7 shows the average CPU time consumed by BBO and TS/BBO algorithm averaged over 100 Monte Carlo runs.
From Table 4, we see that TS/BBO performed the best (on average) on four of the 5 groups. Table 5 shows that BBO was the worst at finding objective function minima when multiple runs are made. Table 6 shows that TS/BBO performed the best on average on all the five groups. Table 7 shows that TS/BBO was a little more effective at finding objective function minima when multiple runs are made, performing the best on 3 of the 5 groups, while BBO performed the best effective on two of the groups. In sum, from Tables 4, 5, 6, and 7, we can draw the conclusion that the more generations, the more populations, and the more elitisms, the smaller objective function value, while the CPU time consumes more.
7.2. TS/BBO versus Population-Based Optimization Method
In order to explore the benefits of TS/BBO, in this section we compared its performance on test-sheet composition problem with seven other population-based optimization methods. ACO (ant colony optimization) [19–21] is an algorithm that is based on the pheromone deposition of ants. DE (differential evolution) [22–25] is a simple method that uses the difference between two solutions to probabilistically adapt a third solution. An ES (evolutionary strategy) [26, 27] is an algorithm that allows more than two parents to reproduce an offspring. A GA (genetic algorithm) [28–32] is a method that is based on natural selection in the theory of biological evolution. PBIL (probability-based incremental learning) [33] is a type of GA that maintains statistics about the population instead of maintaining the population directly. PSO (particle swarm optimization) [34] is based on the swarming behavior of birds, fish, and other creatures. A stud genetic algorithm (SGA) [35] is a GA that uses the best individual at each generation for crossover.
We did some fine tuning on each of the optimization algorithms to get optimal performance, to get the optima for every algorithm. For ACO, we used the following parameters: initial pheromone value , pheromone update constant , exploration constant , global pheromone decay rate , local pheromone decay rate , pheromone sensitivity , and visibility sensitivity . For TS/BBO, we used the parameters presented in Section 7.1. For DE, we used a weighting factor and a crossover constant . For the ES, we produced offspring for each generation and standard deviation for changing solutions. For the GA, we used roulette wheel selection, single-point crossover with a crossover probability of 1, and a mutation probability of 0.01. For PBIL, we used a learning rate of 0.05, 1 good population member, 0 bad population members to use to update the probability vector each generation, and a 0 probability vector mutation rate. For PSO, we used only global learning (no local neighborhoods), an inertial constant = 0.3, a cognitive constant = 1, and a social constant = 1 for swarm interaction. For the SGA, we used single-point crossover with a crossover probability of 1, and a mutation probability of 0.01. Note that the algorithm DE has not the parameter Keep, so we cannot compare Keep between TS/BBO and DE.
To compare the different effects among the parameters Popsize, Maxgen, and Keep (the number of elitism), we ran 100 Monte Carlo simulations of each algorithm on the above problem to get representative performances. Tables 8, 9, 10, and 11 show the results of the simulations. Tables 8, 9, and 10 show the best, worst, and average performance of each algorithm over 100 Monte Carlo runs, respectively, while Table 11 shows the average CPU time consumed by each algorithm over 100 Monte Carlo runs.
From Table 8, we see that TS/BBO performed the best (on average) on the five groups. Table 9 shows that PBIL was the worst at finding objective function minima on all the five groups when multiple runs are made, while the SGA was the best on three of the groups and TS/BBO was the best on two of the groups in the worst values. Table 10 shows that TS/BBO was the most effective at finding objective function minima when multiple runs are made, performing the best on 3 of the 5 groups, while SGA was the second most effective, performing the best on 2 of the 5 groups. Table 11 shows that TS/BBO was the most effective at finding objective function minima when multiple runs are made, performing the best on all the 5 groups. In sum, from Tables 8, 9, 10, and 11 we can draw the conclusion that the more generations, the more populations, and the more elitisms, the smaller objective function value, while the CPU time consumes more. From Table 8 groups 2 and 3 (i.e., row 2 and row 3), we can arrive at a conclusion that the objective function value is not always better when Popsize is increasing. From Table 8 groups 1 and 2 (i.e., row 2 and row 3) or groups 4 and 5 (i.e., row 4 and row 5), we can come to a conclusion that the more Keep, the better results when other parameters are the same. Also, From Table 8 we can reach a decision that increasing Maxgen can get more effective than increasing Popsize.
The simulation implemented in this section shows that the algorithm TS/BBO that we proposed performed the best and most effectively, and it can solve the test-sheet problem perfectly.
8. Conclusion
To improve performance for test-sheet composition, we combined the advantage of tabu search and biogeography-based optimization and proposed a new algorithm TS/BBO. Simulation experiment demonstrates that AHP and TS/BBO that we proposed for test-sheet composition optimization problem have the following advantages.(a)Speed up extracting questions and compute the objective function after preprocessing the item bank according to question type.(b)Habitat encoding length is equal to the total number of testsheet, saving storage space and reducing the optimization space, to ensure that the constraint for total score is met and improve the solution accuracy.(c)We use subencoding according to the question type to ensure that constraint for question type score is satisfied, reducing the optimization space and improving the solution accuracy.(d)AHP determines the test-sheet composition weights, comprehensively considering the objective and subjective factors, in line with the actual test environment, fitting real test needs.(e)Tabu search optimizes mutation and migration operator to create the algorithm TS/BBO, improving solution efficiency and solution accuracy.
However, the algorithm TS/BBO that we proposed in this paper has the following disadvantages: the preprocessing time increases when the number of questions in item bank is getting bigger, which will affect the entire test-sheet composition speed; need further optimization to improve accuracy; need for further ease the conflict between expanding population diversity and reducing the optimization space. The above problems are worth further study.