- About this Journal
- Abstracting and Indexing
- Aims and Scope
- Annual Issues
- Article Processing Charges
- Articles in Press
- Author Guidelines
- Bibliographic Information
- Citations to this Journal
- Contact Information
- Editorial Board
- Editorial Workflow
- Free eTOC Alerts
- Publication Ethics
- Reviewers Acknowledgment
- Submit a Manuscript
- Subscription Information
- Table of Contents
Mathematical Problems in Engineering
Volume 2012 (2012), Article ID 897027, 19 pages
New Bounds for Ternary Covering Arrays Using a Parallel Simulated Annealing
1Instituto Tecnológico Superior de Salvatierra, Madero 303, 38900 Salvatierra, Guanajuato, Mexico
2Information Technology Laboratory, Cinvestav Tamaulipas, Km. 5.5 Carretera Victoria-Soto La Marina, 87130 Victoria, TAMPS, Mexico
3Instituto de Instrumentación para Imagen Molecular (I3M), Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain
Received 24 February 2012; Revised 15 June 2012; Accepted 7 July 2012
Academic Editor: John Gunnar Carlsson
Copyright © 2012 Himer Avila-George et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
A covering array (CA) is a combinatorial structure specified as a matrix of N rows and k columns over an alphabet on v symbols such that for each set of t columns every t-tuple of symbols is covered at least once. Given the values of t, k, and v, the optimal covering array construction problem (CAC) consists in constructing a CA (N; t, k, v) with the minimum possible value of N. There are several reported methods to attend the CAC problem, among them are direct methods, recursive methods, greedy methods, and metaheuristics methods. In this paper, There are three parallel approaches for simulated annealing: the independent, semi-independent, and cooperative searches are applied to the CAC problem. The empirical evidence supported by statistical analysis indicates that cooperative approach offers the best execution times and the same bounds as the independent and semi-independent approaches. Extensive experimentation was carried out, using 182 well-known benchmark instances of ternary covering arrays, for assessing its performance with respect to the best-known bounds reported previously. The results show that cooperative approach attains 134 new bounds and equals the solutions for other 29 instances.
A covering array, denoted by , is a matrix of size which takes values from the set of symbols (called the alphabet), and every submatrix of size contains each tuple of symbols of size (or -tuple), at least once. The value is the number of rows of , is the number of parameters, where each parameter can take values, and the interaction degree between parameters is described by the strength . Each combination of columns must cover all the -tuples. Given that there are sets of columns in , the total number of -tuples in must be . When a -tuple is missing in a specific set of columns, we will refer to it as a missing -wise combination. Then, is a covering array if the number of missing -wise combinations is zero.
When a matrix has the minimum possible value of to be a , the value is known as the covering array number. The covering array number is formally defined as . Given the values of , , and , the optimal covering array construction problem (CAC) consists in constructing a such that the value of is minimized.
A major application of covering arrays (CAs) arises in software interaction testing , where a covering array can be used to represent an interaction test suite as follows. In a software test, we have components or factors. Each of these parameters has values or levels. A test suite is an array where each row is a test case. Each column represents a component, and a value in the column is the particular configuration. By mapping a software test problem to a covering array of strength , we can guarantee that we have tested, at least once, all -way combinations of component values . Thus, software testing costs can be substantially reduced by minimizing the number of test cases in the covering array. Please observe that software interaction testing is a black-box testing technique, and thus it exhibits weaknesses that should be addressed by employing white-box testing techniques. For a detailed example of the use of covering arrays in software interaction testing, the reader is referred to .
In this paper, we aim to develop an enhanced sequential simulated annealing (ESSA) algorithm for finding near-optimal covering arrays. Simulated annealing algorithm is a general-purpose stochastic optimization technique that has proved to be an effective tool for approximating globally optimal solutions to many optimization problems. However, one of the major drawbacks of the technique is the time it requires to obtain good solutions (moreover, when the evaluation function requires too much time). To address this drawback, we propose three parallel simulated annealing approaches to solve the CAC problem. The objective is to find the best bounds to some ternary covering arrays by using parallelism. To our knowledge, the application of parallel simulated annealing to the CAC problem has not been reported in the literature. Some methods of parallelization of simulated annealing are discussed in [4–8].
The remainder of this paper is organized as follows. Some techniques that have been used for constructing covering arrays are presented in Section 2. Section 3 describes the components of our sequential annealing algorithm. In Section 4, three parallel simulated annealing approaches are discussed. Section 5 describes the experimental results. Finally, Section 6 presents the conclusions derived from the research presented in this paper.
2. Review of Covering Arrays Construction Methods
Because of the importance of the construction of (near) optimal covering arrays, much research has been carried out in developing effective methods for constructing them. There are several reported methods for constructing these combinatorial models. Among them are direct methods; recursive methods; greedy methods; metaheuristics methods.
Chateauneuf and Kreher  introduced a new method to construct covering arrays of strength three. This construction uses the structure of covering arrays and the repetition in covering arrays. The idea is to construct a covering array from a small array, a starter vector, and a group. This construction builds the covering array column by column by considering the group acting on the columns of the starter vector. Meagher and Stevens  extended the idea exhibited in , presenting a strategy for obtaining the starter vector by local search and the selection of a group action. Recently, Lobb et al.  presented a generalization of this method to permit any number of fixed points, permit an arbitrary group acting on the symbols, and permit an arbitrary group acting on the columns. With all these generalizations were obtained new bounds for covering arrays of strength two.
Tang and Woo  used constant weight vectors to construct test suites to be applied to logic circuit testing. Martinez-Pena et al.  introduced a new method for constructing covering arrays using trinomial coefficients, and they improved the results presented in . Martinez-Pena et al. used the trinomial coefficients for the representation of the search space in the construction of ternary covering arrays. It is clear that any covering array is formed by a row set. In this sense, a trinomial coefficient represents a particular subset of rows which may belong to a ternary covering array.
Hartman  presented a recursive construction which gives a method of squaring the number of columns in a covering array of strength while multiplying the rows by a factor dependent only on and , but independent of . This factor is related to the Turan numbers that are defined to be the number of edges in the Turan graph.
Colbourn et al.  presented a product construction for . In general, the product of two covering arrays where consists in obtaining a new covering array where the number of columns is equal to the product of the columns of the ingredients, and the number of rows is equal to the sum of the rows of each ingredient.
The majority of commercial and open-source test data generating tools use greedy algorithms for covering arrays construction (AETG, TCG, ACTS, DDA, among others).
Cohen et al.  presented a strategy called AETG. In AETG, covering arrays are constructed one row at a time. To generate a row, the first -tuple is selected based on the one involved in most uncovered pairs. Remaining factors are assigned levels in a random order. Levels are selected based on the one that covers the most new -tuples. For each row that is actually added to the covering array, there are a number of, , candidate rows that are generated, and only a candidate that covers the most new -tuples is added to the covering array. Once a covering is constructed, a number, , of test suites are generated and the smallest test suite generated is reported. This process continues until all pairs are covered.
Tung and Aldiwan  proposed a tool called TCG. In TCG, one row is added at a time to a covering array until all pairs are covered. Before each row is added, a number of up to candidate rows are generated, and the best candidate (covering the most new pairs) is added. is defined to be the maximum cardinality of factors (the maximum number of levels associated with any factor). To construct each row, factors are assigned levels in an order based on a nonascending order of the cardinality of each factor. Each level for the factor is evaluated, and a count of the number of pairs that are covered is used to determine whether or not to select a level for a factor.
Bryce and Colbourn  presented an algorithm called DDA. The DDA constructs one row of a covering array at a time using a steepest ascent approach. Factors are dynamically fixed one at a time in an order based on density. New rows are continually added until all interactions have been covered.
Lei and Tai  introduced a new algorithm called IPO, for pairwise testing. For a system with two or more input parameters, the IPO strategy generates a pairwise test set for the first two parameters, extends the test set to generate a pairwise test set for the first three parameters, and continues to do so for each additional parameter. Contrary to many other algorithms that build covering arrays one row at a time, the IPO strategy constructs them one column at a time. Lei et al.  introduced an algorithm for the efficient production of covering arrays, called IPOG, which generalizes the IPO strategy from pairwise testing to multiway testing. The main idea is that covering arrays of columns can be used to efficiently build a covering array with degree .
Ronneseth and Colbourn  introduced a new algorithm for constructing covering arrays, the BBA. The BBA's fundamental idea is to combine smaller covering arrays by reordering the rows and then to append additional rows for the remaining uncovered pairs.
Some stochastic algorithms in artificial intelligence, such as tabu search [20, 21], simulated annealing , generic algorithms, and ant colony optimization algorithm , provide an effective way to find approximated solutions. In these algorithms, the optimization focuses on one value of at a time, attempting to find a covering array for that size.
A simulated annealing metaheuristic (henceforth called SAC) has been applied by Cohen et al. in  for constructing covering arrays. SAC starts with a randomly generated initial solution whose cost is measured as the number of uncovered -tuples. In their implementation, Cohen et al. use a simple geometric function with an initial temperature fixed at . At each temperature, 2000 neighboring solutions are generated. The algorithm stops either if a valid covering array is found, or if no change in the cost of the current solution is observed after 500 trials. The authors justify their choice of these parameter values based on some experimental tuning. They conclude that their simulated annealing implementation is able to produce smaller covering arrays than other computational methods, sometimes improving upon algebraic constructions.
Torres-Jimenez and Rodriguez-Tello  introduced a new simulated annealing implementation for constructing binary covering arrays.
3. Sequential Simulated Annealing
In this section, we briefly review simulated annealing (SA) algorithm and propose an enhanced sequential simulated annealing (ESSA) implementation to solve the CAC problem. ESSA is an extension of the simulated annealing presented in  for constructing covering arrays for and mixed-level covering arrays.
Simulated annealing is a randomized local search method based on the simulation of annealing of metal. The acceptance probability of a trial solution is given by (3.1), where is the temperature of the system, is the difference of the costs between the trial and the current solutions (the cost change due to the perturbation), and (3.1) means that the trial solution is accepted by nonzero probability even though the solution deteriorates (uphill move)
Uphill moves enable the system to escape from the local minima; without them, the system would be trapped into a local minimum. Too high of a probability for the occurrence of uphill moves, however, prevents the system from converging. In simulated annealing, the probability is controlled by temperature in such a manner that at the beginning of the procedure the temperature is sufficiently high, in which a high probability is available, and as the calculation proceeds, the temperature is gradually decreased, lowering the probability .
The following paragraphs will describe each of the components of the implementation of our simulated annealing implementation. The description is done given the matrix representation of a covering array.
3.1. Internal Representation
Let be a potential solution in the search space , that is, a covering array of size , strength , degree , and order . Then is represented as an array on symbols, in which the element denotes the symbol assigned in the test configuration to the parameter . The size of the search space is then given by
3.2. Initial Solution
The initial solution is constructed by generating as a matrix with maximum Hamming distance. The Hamming distance between two rows is the number of elements in which they differ. Let be a row of the matrix . To generate a random matrix of maximum Hamming distance, the following steps are performed: (1)generate the first row at random,(2)generate rows at random, which will be candidate rows, (3)select the candidate row that maximizes the Hamming distance according to (3.3) and added to the th row of the matrix , (4)repeat from step 2 until is completed;
An example is shown in (3.4); the number of symbols different between rows and is 4 and between and is 3 summing up 7. Then, the Hamming distance for the candidate row is 7:
Equation (3.4) is an example of the Hamming distance between two rows , that are already in the matrix and a candidate row .
3.3. Evaluations Function
The evaluation function is used to estimate the goodness of a candidate solution. Previously reported metaheuristic algorithms for constructing covering arrays have commonly evaluated the quality of a potential solution (covering array) as the number of combination of symbols missing in the matrix [20, 22, 23]. Then, the expected solution will be zero missing. In the proposed simulated annealing implementation, this evaluation function definition was used. Its computational complexity is equivalent to .
3.4. Neighborhood Function
Given that ESSA is based on local search (LS), then a neighborhood function must be defined. The main objective of the neighborhood function is to identify the set of potential solutions which can be reached from the current solution in a local search (LS) algorithm. In case two or more neighborhoods exhibit complementary characteristics, it is then possible and interesting to create more powerful compound neighborhoods. The advantage of such an approach is well documented in . Following this idea, and based on the results of our preliminary experimentations, a neighborhood structure composed by two different functions is proposed for ESSA.
The neighborhood function makes a random search of a missing -tuple and then tries by setting the th combination of symbols in every row of . The neighborhood function randomly chooses a position of the matrix and makes all possible changes of symbol. During the search process, a combination of both and neighborhood functions is employed by ESSA algorithm. The former is applied with probability , while the latter is employed at a rate. This combined neighborhood function is defined in (3.5), where is a random number in the interval :
3.5. Cooling Schedule
The cooling schedule determines the degree of uphill movement permitted during the search and is thus critical to the ESSA algorithm's performance. The parameters that define a cooling schedule are an initial temperature, a final temperature or a stopping criterion, the maximum number of neighboring solutions that can be generated at each temperature, and a rule for decrementing the temperature. The literature offers a number of different cooling schedules, see, for instance, [4, 27]. ESSA uses a geometrical cooling scheme mainly for its simplicity. It starts at an initial temperature which is decremented at each round by a factor using the relation . For each temperature, the maximum number of visited neighboring solutions is . It depends directly on the parameters (, , and ) of the studied covering array. This is because more moves are required for covering arrays with alphabets of greater cardinality.
3.6. Termination Condition
The stop criterion for ESSA is either when the current temperature reaches , when it ceases to make progress, or when a valid covering array is found. In the proposed implementation, a lack of progress exists if after (frozen factor) consecutive temperature decrements the best-so-far solution is not improved.
3.7. Simulated Annealing Pseudocode
Algorithm 1 presents the simulated annealing heuristic as described above. The meaning of the three functions is obvious: INITIALIZE computes a start solution and initial values of the parameters and ; GENERATE selects a solution from the neighborhood of the current solution, using the neighborhood function ; CALCULATE_CONTROL computes a new value for the parameter (cooling schedule) and the number of consecutive temperature decrements with no improvement in the solution.
This is unlike the classical method which takes as a solution to the problem the last value obtained in the annealing chain . We memorize the best solution found during the whole annealing process (see lines 2 and 12).
In the next section, it is presented three parallel simulated annealing approaches for solving the CAC problem.
4. Parallel Simulated Annealing
Parallelization is recognized like a powerful strategy to increase algorithms efficiency; however, simulated annealing parallelization is a hard task because it is essentially a sequential process.
In evaluating performance of a parallel simulated annealing (PSA), it needs to consider solution quality as well as execution speed. The execution speed may be quantified in terms of speed-up and efficiency . The is defined as the ratio of the execution time (on one processor) by the sequential simulated annealing to that by the PSA (on processors) for an equivalent solution quality. In the ideal case, would be equal to . The is defined as the ratio of the actual to the ideal .
Next, we propose three parallel implementations of the simulated annealing algorithm described in Section 3. For these cases, let denote the number of processors and the length of Markov chain.
4.1. Independent Search Approach
A common approach to parallelizing simulated annealing is the ISA [4, 6, 29]. In this approach, each processor independently perturbs the configuration, evaluates the cost, and decides on the perturbation. The processors carry out the independent annealing searches using the same initial solution and cooling schedule as in the sequential algorithm. At each temperature, executes annealing steps. When each processor finishes, it sends its results to processor . Finally, processor chooses the final solution among the local solutions.
We have implemented a simulated annealing algorithm using ISA approach for constructing covering arrays. In the developed implementation, the processors do not interact during individual annealing processes until all processors find their final solution. Then, the best of the solutions is saved and the others are discarded.
4.2. Semi-Independent Search Approach
Aarts and van Laarhoven  introduced a new parallel simulated annealing algorithm named division algorithm. In the division algorithm, the number of iterations at each temperature is divided equally between the processors. After a change in temperature, each processor may simply start from the final solution obtained by that processor at the previous temperature. The best solution from all the processors is then taken to be the final solution. Another variant of this approach is to communicate the best solution from all the processors to each processor every time the temperature changes. Aarts and van Laarhoven  found no significant differences in the performance of these two variants.
We have developed an implementation of division algorithm; we named the implementation SSA. In SSA, parallelism is obtained by dividing the effort of generation a Markov chain over the available processors. A Markov chain is divided into subchains of the length . In this approach, the processors exchange local information including intermediate solutions and their costs. Then, each processor restarts from the best intermediate ones.
Compared to the ISA, communication overhead in this SSA approach would be increased. However, each processor can utilize the information from other processors such that the decrease in computations and idle times can be greater than the increase in communication overhead. For instance, a certain processor which is trapped in an inferior solution can recognize its state by comparing it with others and may accelerate the annealing procedure. That is, processors may collectively converge to a better solution.
4.3. Cooperative Search Approach
In order to improve the performance of the SSA approach, we propose the cooperative search approach (CSA); it used asynchronous communication among processors accessing the global state to eliminate the idle times. Each processor follows a separate search path, accesses the global state which consists of the current best solution and its cost whenever it finished a Markov subchain, and updates the state if necessary. Once a processor gets the global state, it proceeds to the next Markov subchain with any delay.
Unlike SSA, CSA has the following characteristics:(i)idle times can be reduced since asynchronous communications overlap a part of the computation; (ii)less communication overhead, an isolated access to the global state is needed by each processor at the end of each Markov subchain; (iii)the probability of being trapped in a local optimum can be smaller. This is because not all the processors start from the same state in each Markov subchain.
5. Experimental Results
This section presents an experimental design and results derived from testing the parallel ISA, SSA, and CSA algorithms described in the Section 4. In order to show the performance of these approaches, three experiments were developed. The first experiment had as purpose to fine-tune the probabilities of the neighborhood functions to be selected. The second experiment had as purpose to compare the three approaches in terms of parallel execution time. Among the three approaches, the CSA approach was the fastest. The third experiment evaluated the quality of the solutions of the CSA approach over a new benchmark proposed in this paper. The results were compared against the best-known solutions reported in the literature to construct covering arrays .
The three parallel approaches were implemented using C language and the message passing interface (MPI) library. The implementations were run on the Tirant supercomputer (The Tirant supercomputer: http://www.uv.es/siuv/cas/zcalculo/res/informa.wiki). Tirant comprises 256 JS20 compute nodes (blades) and 5 p515 servers. Every blade has two IBM Power4 processors at 2.0 GHz running Linux operating system with 4 GB of memory RAM and 36 GB of local disk storage. All the servers provide a total of nearly 10 TB of disk storage accessible from every blade through GPFS (global parallel file system). Tirant has in total 512 processors, 1 TB of memory RAM, and 19 TB of disk storage. The following parameters were used for all simulated annealing implementations:(1)Initial temperature ,(2)Final temperature ,(3)Cooling factor ,(4)Maximum neighboring solutions per temperature ,(5)Frozen factor ;(6)The neighborhood function is applied using a probability .
5.1. Tuning of the Parameters of Parallel Simulated Annealing
It is well known that the performance of a simulated annealing algorithm is sensitive to parameter tuning. In this sense, we follow a methodology for a fine-tuning of the two neighborhood functions used in ESSA. The fine-tuning was based on the linear Diophantine equation (5.1), where represents a neighborhood function, and its value set to 1, is a value in that represents the probability of executing , and is set to 1.0 which is the maximum probability of executing any :
A solution to the given linear Diophantine equation must satisfy (5.2). This equation has 11 solutions, each solution is an experiment that tests the degree of participation of each neighborhood function in ESSA to accomplish the construction of a covering array. Every combination of the probabilities was applied by ESSA to construct the set of covering arrays shows in Table I(a), and each experiment was run 31 times; with the data obtained for each experiment, we calculate the median. A summary of the performance of ESSA with the probabilities that solved the 100% of the runs is shown in Table I(b):
Finally, given the results shown in Figure 1, the best configuration of probabilities was and because it found the covering arrays in smaller time (median value). The values and were kept fixed in the second experiment.
5.2. Comparison of the ISA, SSA, and CSA Approaches
To test the performance of the ISA, SSA, and CSA approaches, we propose the construction of a covering array with , , , and . Each approach was executed 31 times (for providing statistical validity to experiment) using .
The performance of the algorithms has been compared based on median speedup as a function of the number of processors; the results are shown in Figure 2.
The ISA approach; had difficulty in handling the large problem instances, it does not scale well. The SSA approach provides reasonable results; however, because it is a synchronous algorithm, the idle and communication times are inevitable. The CSA approach is the one which offers the best results; it reduces the execution time of the SSA approach by employing asynchronous information exchange.
In the next subsection, it is presented the third experiment of this work, and the purpose is to measure the performance of the CSA algorithm against the best-known solutions reported in the literature.
5.3. Comparison with State-of-the-Art Procedures
The purpose of this experiment is to carry out a performance comparison of the bounds achieved by the CSA approach with respect to the best-known solutions reported in the literature , which were produced using the following state-of-the-art procedures: orthogonal array construction, Roux type constructions, doubling constructions, algebraic constructions, deterministic density algorithm (DDA), Tabu search, and IPOG-F.
For this experiment, we have fixed the maximum computational time expended by our PSA for constructing a covering array to 72 hours and 50 processors. We create a new benchmark composed of 182 covering arrays distributed as follows:(i)47 covering arrays with strength , degree , and order ,(ii)46 covering arrays with strength , degree , and order ,(iii)45 covering arrays with strength , degree , and order ,(iv)44 covering arrays with strength , degree , and order ,
The detailed results produced by this experiment are listed in Table 2. The first two columns in each subtable indicate the strength and the degree of the selected instances. Next two columns show, in terms of the size of the covering arrays, the best-known solution reported in the literature and the improved bounds produced by the CSA approach. Last column depicts the difference between the best result produced by CSA approach and the best-known solution ().
Figure 3 compares the results shown in Table 2 involving the CSA algorithm and the best-known solutions. The analysis of the data presented led us to the following observation. The solutions quality attained by the CSA approach is very competitive with respect to that produced by the state-of-the-art procedures summarized in column 3 (). In fact, it is able to improve on 134 previous best-known solutions.
The long execution time of simulated annealing due to its sequential nature hinders its application to realistically sized problems, in this case, the CAC problem when , and . A more efficient way to reduce execution time and make the simulated annealing a more promising method is to parallelize sequential simulated annealing. It is a challenging task. In fact, there are many approaches that may be considered in parallelizing simulated annealing. However, an inappropriate strategy used will likely result in poor performance.
In this paper, we have used three different approaches to do the parallelization of the simulated annealing algorithm. From the experimental results, we found that the ISA approach has the worst performance, and it does not scale well. The SSA approach offers reasonable execution times; compared to the ISA, communication overhead in the SSA approach would be increased when the size of the problem grows. However, each processor can utilize the information from other processors such that the decrease in computations and idle times can be greater than the increase in communication overhead. For instance, a certain processor which is trapped in an inferior solution can recognize its state by comparing it with others and may accelerate the annealing procedure. That is, processors may collectively converge to a better solution.
The CSA approach is the one which offers the best results; it significantly reduces the execution time of the SSA approach by employing asynchronous information exchange. The quality solutions attained by the CSA approach are very competitive with respect to that produced by the state-of-the-art procedures; in fact, it is able to improve on 134 previous best-known solutions and equals the solutions for other 29 instances.
These experimental results confirm the practical advantages of using CSA algorithm in the software testing area. It is a robust algorithm yielding smaller test suites than other representative state-of-the-art algorithms, which allows reducing software testing costs.
The new bounds are available in Cinvestav covering array repository (CAR), which is available under request at http://www.tamps.cinvestav.mx/~jtj/authentication.php. We have verified all covering arrays described in this paper using the tool described in .
The authors thankfully acknowledge the computer resources and assistance provided by Spanish Supercomputing Network (TIRANT-UV). This research work was partially funded by the following projects: CONACyT 58554; Cálculo de Covering Arrays; 51623-Fondo Mixto CONACyT; Gobierno del Estado de Tamaulipas.
- D. M. Cohen, S. R. Dalal, J. Parelius, and G. C. Patton, “The combinatorial design approach to automatic test generation,” IEEE Software, vol. 13, no. 5, pp. 83–88, 1996.
- K. Z. Zamli, M. F. J. Klaib, M. I. Younis, N. A. M. Isa, and R. Abdullah, “Design and implementation of a t-way test data generation strategy with automated execution tool support,” Information Sciences, vol. 181, no. 9, pp. 1741–1758, 2011.
- A. Hartman, “Software and hardware testing using combinatorial covering suites,” in Graph Theory, Combinatorics and Algorithms, M. C. Golumbic and I. B. A. Hartman, Eds., vol. 34 of Operations Research/Computer Science Interfaces, pp. 237–266, Springer, New York, NY, USA, 2005.
- E. H. L. Aarts and P. J. M. van Laarhoven, “Statistical cooling: a general approach to combinatorial optimization problems,” Philips Journal of Research, vol. 40, no. 4, pp. 193–226, 1985.
- D. Janaki Ram, T. H. Sreenivas, and K. G. Subramaniam, “Parallel simulated annealing algorithms,” Journal of Parallel and Distributed Computing, vol. 37, no. 2, pp. 207–212, 1996.
- S. Y. Lee and K. G. Lee, “Synchronous and asynchronous parallel simulated annealing with multiple Markov chains,” IEEE Transactions on Parallel and Distributed Systems, vol. 7, no. 10, pp. 993–1008, 1996.
- D. J. Chen, C. Y. Lee, C. H. Park, and P. Mendes, “Parallelizing simulated annealing algorithms based on high-performance computer,” Journal of Global Optimization, vol. 39, no. 2, pp. 261–289, 2007.
- Z. J. Czech, W. Mikanik, and R. Skinderowicz, “Implementing a parallel simulated annealing algorithm,” in Proceedings of the 8th International Conference on Parallel Processing and Applied Mathematics (PPAM'10), vol. 6067 of Lecture Notes in Computer Science, pp. 146–155, Springer, Berlin, Germany, 2010.
- M. Chateauneuf and D. L. Kreher, “On the state of strength-three covering arrays,” Journal of Combinatorial Designs, vol. 10, no. 4, pp. 217–238, 2002.
- K. Meagher and B. Stevens, “Group construction of covering arrays,” Journal of Combinatorial Designs, vol. 13, no. 1, pp. 70–77, 2005.
- J. R. Lobb, C. J. Colbourn, P. Danziger, B. Stevens, and J. Torres-Jimenez, “Cover starters for covering arrays of strength two,” Discrete Mathematics, vol. 312, no. 5, pp. 943–956, 2012.
- D. T. Tang and L. S. Woo, “Exhaustive test pattern generation with constant weight vectors,” IEEE Transactions on Computers, vol. C-32, no. 12, pp. 1145–1150, 1983.
- J. Martinez-Pena, J. Torres-Jimenez, N. Rangel-Valdez, and H. Avila-George, “A heuristic approach for constructing ternary covering arrays using trinomial coefficients,” in Proceedings of the 12th Ibero-American Conference on Artificial Intel-Ligence—IBERAMIA, vol. 6433 of Lecture Notes in Computer Science, pp. 572–581, Springer, Berlin, Germany, 2010.
- C. J. Colbourn, S. S. Martirosyan, G. L. Mullen, D. Shasha, G. B. Sherwood, and J. L. Yucas, “Products of mixed covering arrays of strength two,” Journal of Combinatorial Designs, vol. 14, no. 2, pp. 124–138, 2006.
- Y. W. Tung and W. S. Aldiwan, “Automating test case generation for the new generation mission software system,” in Proceedings of the IEEE Aerospace Conference, vol. 1, pp. 431–437, IEEE Press, March 2000.
- R. C. Bryce and C. J. Colbourn, “The density algorithm for pairwise interaction testing,” Software Testing Verification and Reliability, vol. 17, no. 3, pp. 159–182, 2007.
- Y. Lei and K. C. Tai, “In-parameter-order: a test generation strategy for pair-wise testing,” in Proceedings of the 3rd IEEE International Symposium on High-Assurance Systems Engineering (HASE'98), pp. 254–261, IEEE Computer Society, 1998.
- Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, “IPOG: a general strategy for T-way software testing,” in Proceedings of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07), pp. 549–556, IEEE Computer Society, Tucson, Ariz, USA, March 2007.
- A. H. Ronneseth and C. J. Colbourn, “Merging covering arrays and compressing multiple sequence alignments,” Discrete Applied Mathematics, vol. 157, no. 9, pp. 2177–2190, 2009.
- K. J. Nurmela, “Upper bounds for covering arrays by tabu search,” Discrete Applied Mathematics, vol. 138, no. 1-2, pp. 143–152, 2004.
- L. Gonzalez-Hernandez, N. Rangel-Valdez, and J. Torres-Jimenez, “Construction of mixed covering arrays of variable strength using a tabu search approach,” in Proceedings of the 4th International Conference on Combinatorial Optimization and Applications (COCOA'10), vol. 6508 of Lecture Notes in Computer Science, pp. 51–64, Springer, Berlin, Germany, 2010.
- M. B. Cohen, C. J. Colbourn, and A. C. H. Ling, “Augmenting simulated annealing to build interaction test suites,” in Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE'03), pp. 394–405, IEEE Computer Society, 2003.
- T. Shiba, T. Tsuchiya, and T. Kikuno, “Using artificial life techniques to generate test cases for combinatorial testing,” in Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC'04), vol. 1, pp. 72–77, IEEE Computer Society, September 2004.
- J. Torres-Jimenez and E. Rodriguez-Tello, “New bounds for binary covering arrays using simulated annealing,” Information Sciences, vol. 185, no. 1, pp. 137–152, 2012.
- Y. Jun and S. Mizuta, “Detailed analysis of uphill moves in temperature parallel simulated annealing and enhancement of exchange probabilities,” Complex Systems, vol. 15, no. 4, pp. 349–358, 2005.
- L. Cavique, C. Rego, and I. Themido, “Subgraph ejection chains and tabu search for the crew scheduling problem,” Journal of the Operational Research Society, vol. 50, no. 6, pp. 608–616, 1999.
- M. M. Atiqullah, “An efficient simple cooling schedule for simulated annealing,” in Proceedings of the International Conference on Computational Science and its Applications (ICCSA'04), vol. 3045 of Lecture Notes in Computer Science, pp. 396–404, Springer, Berlin, Germany, 2004.
- S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science, vol. 220, no. 4598, pp. 671–680, 1983.
- Z. Czech, “Three parallel algorithms for simulated annealing,” in Parallel Processing and Applied Mathematics, vol. 2328 of Lecture Notes in Computer Science, pp. 210–217, Springer, Berlin, Germany, 2006.
- C. J. Colbourn, “Covering array tables for t=2, 3, 4, 5, 6,” 2012, http://www.public.asu.edu/~ccolbou/src/tabby/catable.html.
- H. Avila-George, J. Torres-Jimenez, N. Rangel-Valdez, A. Carrión, and V. Hernández, “Supercomputing and a Grid computing on the verification of covering arrays,” The Journal of Supercomputing. In press.