Test-Sheet Composition Using Analytic Hierarchy Process and Hybrid Metaheuristic Algorithm TS/BBO

Abstract

Due to the shortcomings in the traditional methods which dissatisfy the examination requirements in composing test sheet, a new method based on tabu search (TS) and biogeography-based optimization (BBO) is proposed. Firstly, according to the requirements of the test-sheet composition such as the total score, test time, chapter score, knowledge point score, question type score, cognitive level score, difficulty degree, and discrimination degree, a multi constrained multiobjective model of test-sheet composition is constructed. Secondly, analytic hierarchy process (AHP) is used to work out the weights of all the test objectives, and then the multiobjective model is turned into the single objective model by the linear weighted sum. Finally, an improved biogeography-based optimization—TS/BBO is proposed to solve test-sheet composition problem. To prove the performance of TS/BBO, TS/BBO is compared with BBO and other population-based optimization methods such as ACO, DE, ES, GA, PBIL, PSO, and SGA. The experiment illustrates that the proposed approach can effectively improve composition speed and success rate.

1. Introduction

Examination is the process of teaching management which plays an important role in evaluating student achievement, inspiring students’ creativity, and improving student learning outcomes. As an important means of testing student learning and teacher teaching, it also provides the necessary information for teachers to improve their teaching methods and the quality of teaching [1]. Also, tests are often used to help instructors assess students’ prior knowledge [2]. With the advent of computer based education (CBE), how to take full advantage of computers to replace manual to help the examination organizer deal with a variety of complex repetitive work is an important topic in computer managed instruction, so computer-aided test (CAT) come into being. Computer-aided test can best satisfy the different evaluation requirements; quickly and effectively complete test-sheet composition with randomness, scientificity and rationality.

Test-sheet composition is a hot issue in computer-aided testing. At present, the algorithms to solve test-sheet composition are abductive machine learning [3], harmony search algorithm [4], genetic algorithm [5, 6], and particle swarm optimization [7]. However, these algorithms has many shortcomings, such as, meeting the local constraints, parameter uncertainty, the lack of random selection and too long time required for composing test-sheet, which lead to dissatisfy evaluation requirements.

Firstly proposed by Simon in 2008, biogeography-based optimization (BBO) [8–10] is a novel powerful metaheuristic algorithm, which has good global optimization capability, insensitivity for selection of initial values and parameters, strong robustness, simplicity, and easy implementation. Therefore, it has been widespread concern in the academic research and application and penetrated to a number of fields [11]. However, in the field of test-sheet composition, no application of BBO algorithm exists yet. In this paper, we use an improved BBO algorithm to solve the test-sheet composition problem. To reduce the search space and improve efficiency, an improved BBO algorithm makes use of subencoding based on question type. Moreover, we use tabu search (TS) to avoid duplication of encoding in the migration and mutation operations. Analytic hierarchy process (AHP) is used to determine the weight of the test-sheet goal, and then multiobjective optimization model is converted to a single-objective model by a linear weighted sum. In [12], Hwang et al. use tabu search to solve a set of parallel test sheets with identical test ability and get a good performance. Here, we combine BBO and TS to construct a new algorithm according to AHP and the principle and feature of TS and BBO; that is, we use TS to optimize the mutation and migration operator in BBO and then design the TS/BBO algorithm to get the most optimal question combination to form a testsheet.

The structure of this paper is organized as follows. Section 2 describes the mathematical model in test-sheet composition problem. In Section 3, the weight in test-sheet composition is determined by AHP. Subsequently, preliminary knowledge of TS and BBO algorithm are introduced in Sections 4 and 5, respectively. Then, an improved BBO algorithm for test-sheet composition is presented in Section 6 and the detailed implementation procedure is also described. The simulation experiments are conducted in Section 7. Finally, Section 8 concludes the paper and discusses the future path of our work.

2. Problem Description

Test-sheet composition model is a multiconstraint, multiobjective optimization problem. In this section, we will describe the mathematical model for test-sheet composition, including constraints and objective function.

2.1. Constraints

Automatic test-sheet composition with computer is searching for questions to meet the requirements from item bank. We determine the encoding for the main properties according to objective analyzing and preestablish the goal state matrix A for all the questions in the item bank: 𝐀_𝑀×𝑁=𝑎₁₁⋯𝑎_1𝑁𝑎⋮⋱⋮_𝑀1⋯𝑎_𝑀𝑁, (2.1) where M is the total number of items in the item bank; 𝑁 is the number of attributes for each item (here, 𝑁=9). For goal state matrix A, each row represents all properties every item, and each column represents a property of the entire item in the item bank. It has not more than 𝑁−1 constraints because 𝑎_𝑖1 (𝑖=1,2,…,𝑀) means item number.

The process of composing testsheet is as follows. Firstly, find the row to meet the objectives and requirements of question combination 𝑥=(𝑥₁,𝑥₂,…,𝑥_𝑝) in the state matrix A according to the default settings or parameters user input, where 𝑥_𝑖=(𝑥_𝑖,1,𝑥_𝑖,2,…,𝑥_{𝑖,𝑙𝑖}) is a vector composed by the question type 𝑖,𝑥_𝑖,𝑗∈[1,𝑀] is the number for question 𝑗 in the type 𝑖,1≤𝑖≤𝑝,1≤𝑗≤𝑙_𝑖, where 𝑝 is the type number of questions in the item bank, 𝑙_𝑖 is the number of the 𝑖th question, and ∑^𝑝_𝑖=1𝑙_𝑖=𝑙, 𝑙 is the total question number in a testsheet. For simplicity, we use Boolean vector 𝐾=(𝐾₁,𝐾₂,…,𝐾_𝑀) instead of 𝑥=(𝑥₁,𝑥₂,…,𝑥_𝑝) to represent the required question combination at last. Here, 𝐾_𝑚 is the state variable for the mth row (i.e., the mth item or question), where, 𝐾_𝑚=1 means the mth row selected, that is, there exists 𝑥_𝑖,𝑗=𝑚; 𝐾_𝑚=0 means the mth row not selected, that is, there does not exist 𝑥_𝑖,𝑗=𝑚. To simplify, without loss of generality, we use 𝐾(𝑖)(1≤𝑖≤𝑀) to represent the 𝑖th question selected or not.

Every objective for test-sheet composition corresponds to a constraint, all the 𝑁−1(9−1=8) test-sheet objectives corresponding to the following 𝑁−1 (eight) constraints. Thus, the mathematical model, that is, the constraints for goal state matrix A, can be constructed as follows:(1) Total score ^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖2=𝑆_𝑃, (2.2)where 𝑆_𝑃 is total score for the test-sheet composed; 𝑎_𝑖2 is score for the 𝑖th question.(2) Test time ^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖3=𝑇_𝐸, (2.3)where 𝑇_𝐸 is the total time completing the test-sheet composed; 𝑎_𝑖3 is estimated time for the completion of the 𝑖th question.(3) Chapter scores ^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖2𝐶𝑗=𝑆^𝑗_𝐶, (2.4)where 𝐶(𝑗)=^{0,𝑗≠𝑎𝑖4}_{1,𝑗=𝑎𝑖4;}𝑆^𝑗_𝐶 is the score of the jth chapter; 𝑎_𝑖4 is the chapter number for the 𝑖th question.(4) Knowledge point score ^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖2𝑂𝑗=𝑆^𝑗_𝑂, (2.5)where 𝑂(𝑗)=^{0,𝑗≠𝑎𝑖5}_{1,𝑗=𝑎𝑖5;}𝑆^𝑗_𝑂 is the score for the jth knowledge point; 𝑎_𝑖5 is the knowledge point number for the 𝑖th question.(5) Question type score ^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖2𝑇𝑗=𝑆^𝑗_𝑇, (2.6)where 𝑇(𝑗)=^{0,𝑗≠𝑎𝑖6}_{1,𝑗=𝑎𝑖6;}𝑆^𝑗_𝑇 is the score for the jth question type; 𝑎_𝑖6 is the question type number for the 𝑖th question. All the questions in item bank are divided into four kinds (i.e., Type 1, Type 2, Type 3, and Type 4);(6) Cognitive level score ^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖2𝐴𝑗=𝑆^𝑗_𝐴, (2.7)where 𝐴(𝑗)=^{0,𝑗≠𝑎𝑖7}_{1,𝑗=𝑎𝑖7;}𝑆^𝑗_𝐴 is the score for the jth cognitive level; 𝑎_𝑖7 is the cognitive level number for the 𝑖th question.(7) Difficulty degree ∑^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖2𝑎_𝑖8𝑆_𝑃=DIF_𝑃, (2.8)where DIF_𝑃 is the difficulty degree of the test-sheet composed; 𝑎_𝑖8 is difficult degree for the 𝑖th question.(8)Discrimination degree ∑^𝑀_𝑖=1𝐾(𝑖)𝑎_𝑖2𝑎_𝑖9𝑆_𝑃=DIS_𝑃, (2.9)where DIS_𝑃 is the discrimination degree of the test-sheet composed; 𝑎_𝑖9 is discrimination degree for the 𝑖th question.

2.2. Objective Function

Through analysis, eight constraints of the test-sheet composition model (total score, test time, chapter score, knowledge point score, question type score, cognitive level score, difficulty degree, and discrimination degree) should be equal to the evaluation requirements or with the minimum error of the evaluation requirements. Thus, the test-sheet composition is a multiconstraint, multiobjective optimization problem. In practical application, we can consider the error between the above eight constraints and evaluation requirements as objective function 𝑓. Therefore, the composed paper is optimal when 𝑓 reaches minimum.

The mathematical model (i.e., objective function) for the test-sheet composition is as follows: min𝑓=²_𝑗=1𝑑⁺_𝑗+𝑑⁻_𝑗×𝜔_𝑗+^ℎ_𝑖=1𝑑⁺_3𝑖+𝑑⁻_3𝑖×𝜔₃+^𝑙_𝑖=1𝑑⁺_4𝑖+𝑑⁻_4𝑖×𝜔₄+^𝑝_𝑖=1𝑑⁺_5𝑖+𝑑⁻_5𝑖×𝜔₅+^𝑞_𝑖=1𝑑⁺_6𝑖+𝑑⁻_6𝑖×𝜔₆+⁸_𝑗=7𝑑⁺_𝑗+𝑑⁻_𝑗×𝜔_𝑗s.t.^𝑀_𝑗=1𝐾𝑗𝑎_𝑗2−𝑑⁺₁+𝑑⁻₁=𝑆_{𝑃𝑀𝑗=1}𝐾𝑗𝑎_𝑗3−𝑑⁺₂+𝑑⁻₂=𝑇_{𝐸ℎ𝑖=1𝑀𝑗=1}𝐾𝑗𝑎_𝑗2𝐶(𝑖)−𝑑⁺_3𝑖+𝑑⁻_3𝑖=^ℎ_𝑖=1𝑆^𝑖_{𝐶𝑙𝑖=1𝑀𝑗=1}𝐾𝑗𝑎_𝑗2𝑂(𝑖)−𝑑⁺_4𝑖+𝑑⁻_4𝑖=^𝑙_𝑖=1𝑆^𝑖_{𝑂𝑝𝑖=1𝑀𝑗=1}𝐾𝑗𝑎_𝑗2𝑇(𝑖)−𝑑⁺_5𝑖+𝑑⁻_5𝑖=^𝑝_𝑖=1𝑆^𝑖_{𝑇𝑞𝑖=1𝑀𝑗=1}𝐾𝑗𝑎_𝑗2𝐴(𝑖)−𝑑⁺_6𝑖+𝑑⁻_6𝑖=^𝑞_𝑖=1𝑆^𝑖_𝐴∑^𝑀_𝑗=1𝐾𝑗𝑎_𝑗2𝑎_𝑗8𝑆_𝑃−𝑑⁺₇+𝑑⁻₇=DIF_𝑃∑^𝑀_𝑗=1𝐾𝑗𝑎_𝑗2𝑎_𝑗9𝑆_𝑃−𝑑⁺₈+𝑑⁻₈=DIS_𝑃𝑑⁺_𝑖,𝑑⁻_𝑖≥0,𝑑⁺_𝑖×𝑑⁻_𝑖()𝑑=0𝑖=1,2,7,8_+𝑖𝑗,𝑑⁻_𝑖𝑗≥0,𝑑⁺_𝑖𝑗×𝑑⁻_𝑖𝑗𝑑=0𝑖=3,𝑗=ℎ_+𝑖𝑗,𝑑⁻_𝑖𝑗≥0,𝑑⁺_𝑖𝑗×𝑑⁻_𝑖𝑗𝑑=0𝑖=4,𝑗=𝑙_+𝑖𝑗,𝑑⁻_𝑖𝑗≥0,𝑑⁺_𝑖𝑗×𝑑⁻_𝑖𝑗𝑑=0𝑖=5,𝑗=𝑝_+𝑖𝑗,𝑑⁻_𝑖𝑗≥0,𝑑⁺_𝑖𝑗×𝑑⁻_𝑖𝑗𝜔=0𝑖=6,𝑗=𝑞_𝑗≥0𝑗=1,2,…,8,⁸_𝑗=1𝜔_𝑗=1. (2.10)

In the above objective function 𝑓, 𝑑⁺_𝑖(𝑖=1,2,7,8), 𝑑⁺_𝑖𝑗(𝑖=3,𝑗=1,2,…,ℎ), 𝑑⁺_𝑖𝑗(𝑖=4,𝑗=1,2,…,𝑙), 𝑑⁺_𝑖𝑗(𝑖=5,𝑗=1,2,…,𝑝), and 𝑑⁺_𝑖𝑗(𝑖=6,𝑗=1,2,…,𝑞) are the positive deviation between the test-sheet property and evaluation requirements, that is, the part beyond the evaluation requirements; 𝑑⁻_𝑖(𝑖=1,2,7,8), 𝑑⁻_𝑖𝑗(𝑖=3,𝑗=1,2,…,ℎ), 𝑑⁻_𝑖𝑗(𝑖=4,𝑗=1,2,…,𝑙), 𝑑⁻_𝑖𝑗(𝑖=5,𝑗=1,2,…,𝑝), and 𝑑⁻_𝑖𝑗(𝑖=6,𝑗=1,2,…,𝑞) are the negative deviation between the test-sheet property and evaluation requirements, that is, the part less than the evaluation requirements. The positive deviation and negative deviation cannot exist at the same time; therefore, their product will be 0, that is, 𝑑⁻×𝑑⁺=0; 𝜔_𝑗(𝑗=1,2,…,8) is weight for test-sheet composition, whose sum is 1.

In practice, the weight 𝜔_𝑗(𝑗=1,2,…,8) has an important impact on the test-sheet composition whether the result satisfies the requirements. Therefore, how to scientifically and reasonably determine the weight of the test-sheet goal is worthy of further study. In the next section, we will use AHP to determine the weights.

3. Determining the Weight in Test-Sheet Composition Using AHP

AHP [13] is an effective combination of qualitative and quantitative analysis of multiobjective evaluation method, proposed by the University of Pittsburgh professor Saaty in the 1970s. Advantages of AHP are easy implementation, simple structure, speed, and robustness. Due to these advantages, it has many real-world applications, such as evaluating course website quality, improving the service quality of e-learning, and evaluating the web-based e-learning system.

3.1. AHP Algorithm

Proposed by Zahedi [14], four main steps are identified in using the AHP to solve a decision-making problem. The detailed algorithm AHP is described in Algorithm 1.

Begin   Step  1: Setting up the decision hierarchy by breaking down the decision problem into a       hierarchy of interrelated decision elements, as a tree containing the overall goal at       the top with many levels of criteria and subcriteria in between and the alternatives       at the bottom   Step  2: Collecting input data by pairwise comparisons of decision elements   Step  3: Using the eigenvalue method to estimate the relative weights of decision elements   Step  4: Aggregating the relative weights of decision elements to arrive at a set of ratings for       the decision alternatives. End.

Algorithm 1 

The algorithm of analytic hierarchy process.

3.2. Determining the Weights Using AHP

AHP can be used to determine goal weight for the test-sheet composition. Integrating the feature of test-sheet composition into the AHP, specific solution process is shown in Algorithm 2 (Figure 1).

Begin   Step  1: Define test-sheet problem and determine test-sheet composition goal   Step  2: Structure the hierarchy from the top through the intermediate levels (criteria on which       subsequent levels depend) to the lowest level (Figure 1).   Step  3: Construct a set of pair-wise comparison matrix C (size 𝑛×𝑛, here 𝑛=8, shown in       (Table 1). The pair-wise comparisons are done in terms of which element dominates       the other.   Step  4: Calculate the sorted weights. Calculate maximum eigenvalue 𝜆max and its eigenvector        𝜔′ for comparison matrix C through 𝐶𝑥=𝜆𝑥, and then normalize 𝜔′ to get the       final weight vector ω (Table 2).   Step  5: Test consistency     Step  5.1: Calculate consistency index CI through CI=(𝜆max−𝑛)/(𝑛−1)     Step  5.2: Find the average Random Index RI through Table 3.     Step  5.3: Calculate consistency ratio CR through CR = CI/RI   Step  6: If CR<0.1, the  comparison matrix C is consistent and return; if CR≥0.1, the       comparison matrix C should be modified, and return Step  4. End.

Algorithm 2 

The algorithm of AHP for test-sheet composition.

Figure 1 

Level tree structure of test-sheet composition.

We calculate consistency ratio CR=0.0822 for the compassion matrix 𝐶 with Algorithm 2 and get the weights as shown in Table 2, so we can draw a conclusion that the compassion matrix 𝐶 has satisfactory consistency.

Table 1 

Comparison matrix C for test-sheet composition.

Table 2 

Weights 𝜔.

Table 3 

Random index RI for different 𝑛.

In the actual exam, test-sheet goal weight is various. AHP can determine different test-sheet goal weight responding to different requirements according to the different compassion matrix C.

4. Biogeography-Based Optimization (BBO)

Biogeography is the study of the migration, speciation, mutation, and extinction of species [15]. Biogeography has frequently been thought of as a process that compels equilibrium in the number of species in islands. However, equilibrium in a system can also be viewed as a minimum-energy configuration, so we see that biogeography can be considered as an optimization process. This idea is further discussed in [16].

Firstly proposed by Simon in 2008, biogeography-based optimization (BBO) is a new evolution algorithm developed for the global optimization [8]. It is inspired by the immigration and emigration of species between islands (or habitats) in search for more compatible islands. Each solution is called a “habitat” (or “island”) with an HSI (habitat suitability index) and represented by an n-dimension real vector. An initial individual of the habitat vector is generated at random. The habitat with a high HSI is considered to be good solution, while the habitat with a low HSI is considered to be bad solution. Low HSI can take in many new good features form the high HSI, so that these low HSI solutions have a relatively high possibility that become high HSI solutions. In BBO, habitat H is a vector of n (suitable index vectors (SIVs)) initialized randomly and then implements migration and mutation operator to achieve the optimal solution. The new candidate solution is generated from the entire habitat in population by using the migration and mutation operators.

In BBO, migration operator can change existing habitat and modify existing solution. Migration is a probabilistic operator that adjusts habitat 𝑋_𝑖. The probability 𝑋_𝑖 modified is proportional to its immigration rate 𝜆_𝑖, and the source of the modified probability from 𝑋_𝑗 is proportional to the emigration rate 𝜇_𝑗. Migration operator is shown in Algorithm 3.

Begin   for  𝑖=1 to NP     Select 𝑋𝑖 with probability based on 𝜆𝑖     if rand(0, 1) < 𝜆𝑖  then      for  𝑗=1 to NP      Select 𝑋𝑗 with probability based on 𝜇𝑗      if rand(0, 1) < 𝜇𝑗  then       Randomly select an SIV 𝜎 from 𝑋𝑗       Replace a random SIV in 𝑋𝑖 with 𝜎      end if      end for     end if   end for End.

Algorithm 3 

The algorithm of habitat migration operator.

Mutation is also a probabilistic operator that randomly modifies habitat SIVs based on the habitat a priori probability of existence. Very high HSI solutions and very low HSI solutions are equally improbable. Medium HSI solutions are relatively probable. The mutation rate 𝑚 is expressed as 𝑚=𝑚_max1−𝑃_𝑠𝑃_max, (4.1) where 𝑚_max is a user-defined parameter.

Additionally, the mutation operator tends to increase the population diversity. Mutation can be described in Algorithm 4. The basic framework of BBO algorithm can be simply described in Algorithm 5. More details about the migration operator, mutation operator, and BBO algorithm can be found in [8] and in the Matlab code [17].

Begin   for  𝑖=1 to NP    Compute the probability 𝑃𝑖    Select SIV 𝑋𝑖(𝑗) with probability based on 𝑃𝑖    if rand(0, 1) < 𝑚𝑖  then     Replace 𝑋𝑖(𝑗) with a randomly generated SIV    end if   end for End.

Algorithm 4 

The algorithm of mutation operator.

Begin     1: Initialization. Set the generation counter 𝐺=1; Initialize the population 𝑃 randomly     and each habitat corresponding to a potential solution to the given problem.   Step  2: Evaluate the fitness for each individual in 𝑃   Step  3: While the termination criteria is not satisfied do     Sort the population from best to worst.     For each habitat, map the HSI to the number of species 𝑆,     Calculate the immigration rate 𝜆𝑖 and the emigration rate 𝜇𝑖 for each individual 𝑋𝑖.     Modify the population with the migration operator shown in Algorithm 3.     Update the probability for each individual.     Mutate the population with the mutation operation shown in Algorithm 4.     Evaluate the fitness for each individual in 𝑃.     Sort the population from best to worst.     𝐺=𝐺+1;   Step  4: end while End.

Algorithm 5 

The algorithm of biogeography based optimization.

5. Tabu Search (TS)

Tabu search (TS) [18] is a metaheuristic framework which takes advantage of its use of adaptive memory strategies to manage simultaneously the variety and intensification searches in the solution space of the optimization problem under way. The TS framework is composed by the following major components.

(1) Solution Configuration
The candidate solution for the optimization problem under way is encoded by a configuration. Relying on the characteristic of the problem, the solution configuration 𝑥=[𝑥₁,𝑥₂,…,𝑥_𝑛], which contains 𝑛 decision variables, could be a vector taking real numbers, binary values, or mixed combinations of them. Initially, a configuration is set randomly within the reasonable ranges of the decision variables.

(2) Move Function
A move function 𝑆(𝑥) is used to guide the solution configuration 𝑥 to explore its local solution space. The step size of the move should be limited in an appropriate range according to the practical problem. An oversized step makes the search to step over a promising neighbor, while an undersized step causes the algorithm to be inefficient. By performing the move function, the solution configuration is turned into a neighboring configuration, that is, 𝑥^′=𝑆(𝑥), where 𝑥^′ is a neighboring configuration of 𝑥.

(3) Neighborhood
The move function defines a neighborhood which bounds the neighboring configurations which are reachable by implementing a move operation to the current solution configuration using 𝑆(𝑥). Generally, the neighborhood of solution 𝑥 is defined by Ω(𝑥)={𝑥^′∣𝑥^′=𝑆(𝑥)}.

(4) Tabu List
A tabu list which stops a recent move to be reversed is kept and modified in the TS framework. Once a move (say, swapping the 𝑖th and the jth entries of 𝑥) is carried out, the inverse of this move (swapping the two entries again) is recorded in the tabu list and is labeled tabu-active. All the tabu moves are excluded from the neighborhood Ω(𝑥), and thus become inaccessible.

(5) Aspiration Criterion
To allow the solution configuration to move to an attractive but tabu neighboring configuration, an aspiration criterion is strategically designed to overrule the tabu status of such a desired move. Aspiration criterion provides a restricted degree of freedom in accepting a tabu move that achieves a threshold of attractiveness.

(6) Stopping Criterion
The stopping criterion depends on the purpose of the problem. There are a number of alternatives such as a minimal solution quality level, a given CPU time limit, a maximal number of iterations between two improvements of the best solution found.

Basically, the TS approach can be summarized as shown in Algorithm 6.

Begin   Step  1: Randomly generate an initial configuration, 𝑥=[𝑥1,𝑥2,…,𝑥𝑛], with an       empty tabu list   Step  2: Repeat the following statements until the stopping criterion is met    Step  2.1: Identify the neighborhood Ω(𝑥)={𝑥′∣𝑥′=𝑆(𝑥)} of current         configuration 𝑥, and consider a subset 𝐻(𝑥) from Ω(𝑥)    Step  2.2: Choose the best (in terms of a moving evaluation score)         configuration 𝑥∗∈𝐻(𝑥) which is either nontabu or is tabu         but satisfying the aspiration criterion    Step  2.3: Update the tabu list and let 𝑥=𝑥∗   Step  3: Output the best configuration found End.

Algorithm 6 

Algorithm of tabu search.

6. Tabu Search/Biogeography-Based Optimization (TS/BBO)

As we all know, the standard TS algorithm is good at exploring the search space and locating the region of global minimum, but it is relatively slow at exploitation of the solution. On the other hand, standard BBO algorithm is usually quick at the exploitation of the solution, though its exploration ability is relatively poor. Therefore, in our work, a hybrid metaheuristic algorithm by integrating tabu search into biogeography-based optimization, so-called TS/BBO, is used to solve the problem of test-sheet composition. The difference between TS/BBO and BBO is that the hybrid migration operator is used to replace the original BBO mutation operator. In this way, this method can explore the new search space by the mutation of the TS algorithm and exploit the population information with the migration of BBO, and therefore it can overcome the lack of the exploitation of the TS algorithm. In the following, we will show the algorithm TS/BBO which is a variety of TS and BBO. Firstly, we describe tabu search migration and mutation operation, and then a mainframe of TS/BBO is shown.

6.1. Tabu Search Migration and Mutation Operation

The critical operator of TS/BBO is the tabu search migration operator, which composes the tabu search with the migration of BBO. In this algorithm, we can find that the migration between the population 𝑋_𝑖 and 𝑋_𝑗 has no repeat because of the use of tabu search. The core idea of the proposed hybrid migration operator is based on two considerations. First, poor solutions can receive many new features from good solutions. Second, the final solution cannot include any repeat SIV. Pseudocode of tabu search migration operation can be described in Algorithm 7.

Begin   for  𝑖=1 to NP     Select 𝑋𝑖 with probability based on 𝜆𝑖     if rand(0, 1) < 𝜆𝑖  then      for  𝑗=1 to NP      Select 𝑋𝑗 with probability based on 𝜇𝑗      ALLOW = 𝑋𝑗      TS = Φ      for 𝑘=1 to 𝐷       if rand(0, 1) < 𝜇𝑗  then        Randomly select an SIV 𝜎 from ALLOW        Replace a random SIV in 𝑋𝑖 with 𝜎        ALLOW = ALLOW∖ {𝜎}        TS=TS∪{𝜎}       end if      end for      end for     end if   end for End.

Algorithm 7 

Tabu search migration operator of BBO.

In the same way, we compose the tabu search and original mutation operator which modifies the habitat with low HSI in order to avoid the repeat to improve the search efficiency. Pseudocode of tabu search mutation operation can be described in Algorithm 8.

Begin   for  𝑖=1 to NP     Compute the probability 𝑃𝑖     Select SIV 𝑋𝑖(𝑗) with probability based on 𝑃𝑖     ALLOW = {1to𝑀}     TS = Φ     if rand(0, 1) < 𝑚𝑖  then      Replace 𝑋𝑖(𝑗) with a randomly generated SIV from ALLOW      ALLOW = ALLOW∖{SIV}      TS=TS∪{SIV}     end if   end for End.

Algorithm 8 

Tabu search mutation operator of BBO.

6.2. Mainframe of TS/BBO

By incorporating above-mentioned tabu search migration and mutation operator into original BBO algorithm, the TS/BBO has been developed as a new algorithm. TS/BBO algorithm is given in Algorithm 9.

Begin   Step  1:  Initialization. Set the generation counter 𝐺=1; and randomly initialize       a population of NP individuals 𝑃, the parameter TS, 𝑀, ALLOW   Step  2: Evaluate the fitness(HSI) for each individual in 𝑃 according to (4.1).   Step  3:  while The halting criteria is not satisfied do       Sort the population from worst to best according to HSI       For each individual, map the HSI to the number of species       Calculate the immigration rate 𝜆𝑖 and the emigration rate 𝜇𝑖 for each       individual 𝑋𝑖       Modify the population with the tabu search migration operator shown in       Algorithm 7       Update the probability for each individual       Mutate the population with the tabu search mutation operator shown in       Algorithm 8       Evaluate the fitness for each individual in 𝑃       Memorize the best solution achieved so far         𝐺=𝐺+1   Step  4: end while End.

Algorithm 9 

Algorithm description of TS/BBO.

6.3. Algorithm TS/BBO for Test-Sheet Composition

In TS/BBO, the standard continuous encoding of TS/BBO cannot be used to solve test-sheet composition directly. In order to apply TS/BBO to test-sheet composition, one of the key issues is to construct a direct relationship between the test-sheet sequences and the vector of individuals in TS/BBO.

6.3.1. Preprocessing

We will do the following preprocessing before the design of TS/BBO algorithm for test-sheet composition(a)We regroup all the questions in item bank according to question type, and then the candidates take the same question type together to form a subset. So all the questions in item bank can be divided into several different subsets, and then all questions will be renumbered.(b)Because question type in the requirements and item bank has the same score, we can calculate the number of questions required for every question type, and then we get the total number of questions in a testsheet.

6.3.2. Encoding

When using TS/BBO to solve test-sheet composition problem, the status code for each habitat represents a candidate solution, that is, a test-sheet composition scheme. Therefore, how to determine the effective habitat status code is a key issue.

Traditional encoding method is as follows: the status code of each habitat is represented by a binary string whose length is the number of questions in total in item bank, and the number “1” indicates that the question corresponding to the number is selected, while number “0” indicates that the question corresponding to the number is not selected; the length of the number “1” indicates the number of questions contained in total in a testsheet. This encoding method is simple and maximizes a random search at most, but it has increased the search space, reducing the search efficiency.

Therefore, an alternative encoding method is proposed in this paper. This method rearranges questions into different subsets according to the different question type. The status code of each habitat is represented by an l-dimensional vector 𝑥=(𝑥₁,𝑥₂,…,𝑥_𝑝). The dimension 𝑙 for vector 𝑥 depends on total questions in a testsheet, where the vector 𝑥_𝑖=(𝑥_𝑖,1,𝑥_𝑖,2,…,𝑥_{𝑖,𝑙𝑖}) is composed by the 𝑖th question type, the integer 𝑥_𝑖,𝑗∈[1,𝑀] means the jth question in the 𝑖th question type; 1≤𝑖≤𝑝,1≤𝑗≤𝑙_𝑖, 𝑝 is the total question in item bank; 𝑙_𝑖 is the number of the 𝑖th question type, and ∑^𝑝_𝑖=1𝑙_𝑖=𝑙, 𝑙 is the total question in some test-sheet composed.

For example, we have seven questions (1.𝑎,2.𝑏,3.𝑐,4.𝑑,5.𝑒,6.𝑓,and7.𝑔) in item bank; questions a and g belong to question type 1, 1 score for each question; questions b, e, and f belong to question type 2, 2 scores for each question; questions c and d belong to question type 3, 3 scores for each question. Questions in item bank will be rearranged and renumbered according to question type and the result is (1.𝑎,2.𝑔,3.𝑏,4.𝑒,5.𝑓,6.𝑐,and7.𝑑). According to the question type requirement {question type 2 : question type 3} = {4 : 3}, we get a total of 3 questions in the testsheet, that is, 𝑀=7. 𝑝=3, 𝑙₁=0, 𝑙₂=2, 𝑙₃=1, and 𝑙=𝑙₁+𝑙₂+𝑙₃=3. So (4, 5, 7) represents a test-sheet composition scheme; that is, it represents a test-sheet containing three questions which are questions e, f, and d.

6.3.3. The Algorithm TS/BBO for Test-Sheet Composition

Improved BBO can adapt to the needs of testsheet, while optimization algorithms can improve the BBO fast search capabilities and increase the search to the global possible optimum solution. HSI in Habitat 𝑖 is represented by the objective function 𝑓(𝑥_𝑖) in test-sheet composition model the smaller the value 𝑓, then the higher HSI in Habitat 𝑖.

Based on the above analysis, the pseudo code of improved BBO-TS/BBO for test-sheet composition is described as shown in Algorithm 10.

Begin   Step  1:  Preprocessing. Preprocess the questions in item bank described in       subsection 6.3.1, and determine 𝐿 which is the total number of questions       contained in a test-sheet   Step  2: Initializing. Set the generation counter 𝐺=1; set the status code of habitat 𝑖       (i.e., SIV) 𝑥𝑖 according to subsection 6.3.2; randomly initialize a       population of NP individuals 𝑃; set the maximum variation rate 𝑚max and       migration rate 𝑝mod; set dimension for the optimization problem 𝐷; set the       maximum capacity of habitat species 𝑆max; set maximum of immigration       function 𝐼 and the maximum of emigration function 𝐸 and the maximum of       elite individuals retained 𝑧; set the parameter TS, 𝑀, ALLOW related to       tabu search   Step  3: Calculating the immigration and emigration rate. Calculating the species       number 𝑆𝑖, the immigration 𝜆(𝑆𝑖) and emigration rate        𝜇(𝑆𝑖)(𝑖=1,2,…,𝑛) corresponding to habitat 𝑖 based on its SIV 𝑓(𝑥𝑖)       according to (2.10)   Step  4: while The halting criteria is not satisfied do       Sort the population from worst to best according to HSI       For each individual, map the HSI to the number of species       Calculate the immigration rate 𝜆𝑖 and the emigration rate 𝜇𝑖 for each         individual 𝑋𝑖         Modify the population with the taboo search migration operator shown in       Algorithm 7       Update the probability for each individual       Mutate the population with the taboo search mutation operator shown in       Algorithm 8       Evaluate the fitness for each individual in 𝑃       Memorize the best solution achieved so far        𝐺=𝐺+1   Step  5: end while End.

Algorithm 10 

Algorithm of TS/BBO for test-sheet composition.

7. Simulation Experiments

In this section, we look at the performance of TS/BBO as compared with BBO and other population-based optimization methods. Firstly, we compare performances between BBO and TS/BBO, and then we compare performances between TS/BBO and other population-based optimization methods such as ACO, DE, ES, GA, PBIL, PSO, and SGA.

To allow a fair comparison of running times, all the experiments were performed on a PC with a Pentium IV processor running at 3.0 GHz, 1 GB of RAM and a hard drive of 160 Gbytes. Our implementation was compiled using MATLAB R2012a (7.14) running under Windows XP. In the following, we will describe the problem we use to test the performance of the TS/BBO.

7.1. TS/BBO versus BBO

We randomly generate an item bank with M (e.g., 𝑀=10000) questions, and then choose some questions to form a testsheet. Its constraints are shown as follows:(1)the total score is 100;(2)the test time is 120 minutes;(3)chapter score {1 : 2 : 3 : 4 : 5 : 6 : 7 : 8 : 9 : 10}={6 : 6 : 8 : 12 : 9 : 12 : 12 : 15 : 10: 10};(4)knowledge point score {1 : 2 : 3 : 4 : 5 : 6 : 7 : 8 : 9 : 10:11:12:13:14:15:16:17:18:19:20} = {3 : 3 : 3 : 3 : 4 : 4 : 6 : 6 : 4 : 5 : 6 : 6 : 6 : 6 : 6 : 9 : 5 : 5 : 5 : 5} (a chapter contains two knowledge points);(5)question type score {Type 1 : Type 2 : Type 3 : Type 4} = {10 : 16 : 24: 50};(6)cognitive level score {Memorizing∶Understanding∶Applying∶Analyzing∶Comprehending∶Evaluating}  =  {10∶15∶20∶30∶15∶10};(7)difficulty degree is 0.4900;(8)discrimination degree is 0.4050.

We get the goal weights for test-sheet composition {total score : test time : chapter score : knowledge point score : questions type score : cognitive level score : difficulty degree : discrimination degree}={0.3423 : 0.1412 : 0.0792 : 0.0319 : 0.2244 : 0.0417 : 0.0970 : 0.0424} according to the Algorithm 2 as shown in Section 3.2.

For BBO and TS/BBO, we used the same following parameters: habitat modification probability = 1, immigration probability bounds per gene = [0,1], step size for numerical integration of probabilities = 1, maximum immigration and migration rates for each island = 1, and mutation probability = 0.005. We must point out that TS/BBO needs to set tabu search table except for the above parameters. To compare the different effects among the parameters Popsize, Maxgen, and Keep (the number of elitisms), we ran 100 Monte Carlo simulations of BBO and TS/BBO algorithm on the above problem to get representative performances. Tables 4, 5, 6, and 7 show the results of the simulations. In other words, Tables 4, 5, and 6 shows the best, worst, and average performance of BBO and TS/BBO algorithm found by BBO and TS/BBO algorithm over 100 Monte Carlo runs, respectively, while Table 7 shows the average CPU time consumed by BBO and TS/BBO algorithm averaged over 100 Monte Carlo runs.

Table 4 

Best normalized optimization results on test-sheet composition problem. The numbers shown are the best results found after 100 Monte Carlo simulations of BBO and TS/BBO algorithm.

Table 5 

Worst normalized optimization results on test-sheet composition problem. The numbers shown are the best results found after 100 Monte Carlo simulations of BBO and TS/BBO algorithm.

Table 6 

Mean normalized optimization results on test-sheet composition problem. The numbers shown are the minimum objective function values found by BBO and TS/BBO algorithm, averaged over 100 Monte Carlo simulations.

Table 7 

Average CPU time on test-sheet composition problem. The numbers shown are the minimum average CPU time (Sec) consumed by BBO and TS/BBO algorithm.

From Table 4, we see that TS/BBO performed the best (on average) on four of the 5 groups. Table 5 shows that BBO was the worst at finding objective function minima when multiple runs are made. Table 6 shows that TS/BBO performed the best on average on all the five groups. Table 7 shows that TS/BBO was a little more effective at finding objective function minima when multiple runs are made, performing the best on 3 of the 5 groups, while BBO performed the best effective on two of the groups. In sum, from Tables 4, 5, 6, and 7, we can draw the conclusion that the more generations, the more populations, and the more elitisms, the smaller objective function value, while the CPU time consumes more.

7.2. TS/BBO versus Population-Based Optimization Method

In order to explore the benefits of TS/BBO, in this section we compared its performance on test-sheet composition problem with seven other population-based optimization methods. ACO (ant colony optimization) [19–21] is an algorithm that is based on the pheromone deposition of ants. DE (differential evolution) [22–25] is a simple method that uses the difference between two solutions to probabilistically adapt a third solution. An ES (evolutionary strategy) [26, 27] is an algorithm that allows more than two parents to reproduce an offspring. A GA (genetic algorithm) [28–32] is a method that is based on natural selection in the theory of biological evolution. PBIL (probability-based incremental learning) [33] is a type of GA that maintains statistics about the population instead of maintaining the population directly. PSO (particle swarm optimization) [34] is based on the swarming behavior of birds, fish, and other creatures. A stud genetic algorithm (SGA) [35] is a GA that uses the best individual at each generation for crossover.

We did some fine tuning on each of the optimization algorithms to get optimal performance, to get the optima for every algorithm. For ACO, we used the following parameters: initial pheromone value 𝜏₀=1𝐸−6, pheromone update constant 𝑄=20, exploration constant 𝑞₀=1, global pheromone decay rate 𝜌_𝑔=0.9, local pheromone decay rate 𝜌_𝑙=0.5, pheromone sensitivity 𝛼=1, and visibility sensitivity 𝛽=5. For TS/BBO, we used the parameters presented in Section 7.1. For DE, we used a weighting factor 𝐹=0.5 and a crossover constant CR=0.5. For the ES, we produced 𝜆=10 offspring for each generation and standard deviation 𝜎=1 for changing solutions. For the GA, we used roulette wheel selection, single-point crossover with a crossover probability of 1, and a mutation probability of 0.01. For PBIL, we used a learning rate of 0.05, 1 good population member, 0 bad population members to use to update the probability vector each generation, and a 0 probability vector mutation rate. For PSO, we used only global learning (no local neighborhoods), an inertial constant = 0.3, a cognitive constant = 1, and a social constant = 1 for swarm interaction. For the SGA, we used single-point crossover with a crossover probability of 1, and a mutation probability of 0.01. Note that the algorithm DE has not the parameter Keep, so we cannot compare Keep between TS/BBO and DE.

To compare the different effects among the parameters Popsize, Maxgen, and Keep (the number of elitism), we ran 100 Monte Carlo simulations of each algorithm on the above problem to get representative performances. Tables 8, 9, 10, and 11 show the results of the simulations. Tables 8, 9, and 10 show the best, worst, and average performance of each algorithm over 100 Monte Carlo runs, respectively, while Table 11 shows the average CPU time consumed by each algorithm over 100 Monte Carlo runs.

Table 8 

Best normalized optimization results on test-sheet composition problem. The numbers shown are the best results found after 100 Monte Carlo simulations of each algorithm.

Table 9 

Worst normalized optimization results on test-sheet composition problem. The numbers shown are the best results found after 100 Monte Carlo simulations of each algorithm.

Table 10 

Mean normalized optimization results on test-sheet composition problem. The numbers shown are the minimum objective function values found by each algorithm, averaged over 100 Monte Carlo simulations.

Table 11 

Average CPU time on test-sheet composition problem. The numbers shown are the minimum average CPU time (Sec) consumed by each algorithm.

From Table 8, we see that TS/BBO performed the best (on average) on the five groups. Table 9 shows that PBIL was the worst at finding objective function minima on all the five groups when multiple runs are made, while the SGA was the best on three of the groups and TS/BBO was the best on two of the groups in the worst values. Table 10 shows that TS/BBO was the most effective at finding objective function minima when multiple runs are made, performing the best on 3 of the 5 groups, while SGA was the second most effective, performing the best on 2 of the 5 groups. Table 11 shows that TS/BBO was the most effective at finding objective function minima when multiple runs are made, performing the best on all the 5 groups. In sum, from Tables 8, 9, 10, and 11 we can draw the conclusion that the more generations, the more populations, and the more elitisms, the smaller objective function value, while the CPU time consumes more. From Table 8 groups 2 and 3 (i.e., row 2 and row 3), we can arrive at a conclusion that the objective function value is not always better when Popsize is increasing. From Table 8 groups 1 and 2 (i.e., row 2 and row 3) or groups 4 and 5 (i.e., row 4 and row 5), we can come to a conclusion that the more Keep, the better results when other parameters are the same. Also, From Table 8 we can reach a decision that increasing Maxgen can get more effective than increasing Popsize.

The simulation implemented in this section shows that the algorithm TS/BBO that we proposed performed the best and most effectively, and it can solve the test-sheet problem perfectly.

8. Conclusion

To improve performance for test-sheet composition, we combined the advantage of tabu search and biogeography-based optimization and proposed a new algorithm TS/BBO. Simulation experiment demonstrates that AHP and TS/BBO that we proposed for test-sheet composition optimization problem have the following advantages.(a)Speed up extracting questions and compute the objective function after preprocessing the item bank according to question type.(b)Habitat encoding length is equal to the total number of testsheet, saving storage space and reducing the optimization space, to ensure that the constraint for total score is met and improve the solution accuracy.(c)We use subencoding according to the question type to ensure that constraint for question type score is satisfied, reducing the optimization space and improving the solution accuracy.(d)AHP determines the test-sheet composition weights, comprehensively considering the objective and subjective factors, in line with the actual test environment, fitting real test needs.(e)Tabu search optimizes mutation and migration operator to create the algorithm TS/BBO, improving solution efficiency and solution accuracy.

However, the algorithm TS/BBO that we proposed in this paper has the following disadvantages: the preprocessing time increases when the number of questions in item bank is getting bigger, which will affect the entire test-sheet composition speed; need further optimization to improve accuracy; need for further ease the conflict between expanding population diversity and reducing the optimization space. The above problems are worth further study.