New Bounds for Ternary Covering Arrays Using a Parallel Simulated Annealing

Abstract

A covering array (CA) is a combinatorial structure specified as a matrix of N rows and k columns over an alphabet on v symbols such that for each set of t columns every t-tuple of symbols is covered at least once. Given the values of t, k, and v, the optimal covering array construction problem (CAC) consists in constructing a CA (N; t, k, v) with the minimum possible value of N. There are several reported methods to attend the CAC problem, among them are direct methods, recursive methods, greedy methods, and metaheuristics methods. In this paper, There are three parallel approaches for simulated annealing: the independent, semi-independent, and cooperative searches are applied to the CAC problem. The empirical evidence supported by statistical analysis indicates that cooperative approach offers the best execution times and the same bounds as the independent and semi-independent approaches. Extensive experimentation was carried out, using 182 well-known benchmark instances of ternary covering arrays, for assessing its performance with respect to the best-known bounds reported previously. The results show that cooperative approach attains 134 new bounds and equals the solutions for other 29 instances.

1. Introduction

A covering array, denoted by CA(𝑁;𝑡,𝑘,𝑣), is a matrix 𝑀 of size 𝑁×𝑘 which takes values from the set of symbols {0,1,2,…,𝑣−1} (called the alphabet), and every submatrix of size 𝑁×𝑡 contains each tuple of symbols of size 𝑡 (or 𝑡-tuple), at least once. The value 𝑁 is the number of rows of 𝑀, 𝑘 is the number of parameters, where each parameter can take 𝑣 values, and the interaction degree between parameters is described by the strength 𝑡. Each combination of 𝑡 columns must cover all the 𝑣^𝑡𝑡-tuples. Given that there are ^𝑘𝑡 sets of 𝑡 columns in 𝑀, the total number of 𝑡-tuples in 𝑀 must be 𝑣^𝑡^𝑘𝑡. When a 𝑡-tuple is missing in a specific set of 𝑡 columns, we will refer to it as a missing 𝑡-wise combination. Then, 𝑀 is a covering array if the number of missing 𝑡-wise combinations is zero.

When a matrix has the minimum possible value of 𝑁 to be a CA(𝑁;𝑡,𝑘,𝑣), the value 𝑁 is known as the covering array number. The covering array number is formally defined as CAN(𝑡,𝑘,𝑣)=min{𝑁∶∃CA(𝑁;𝑡,𝑘,𝑣)}. Given the values of 𝑡, 𝑘, and 𝑣, the optimal covering array construction problem (CAC) consists in constructing a CA(𝑁;𝑡,𝑘,𝑣) such that the value of 𝑁 is minimized.

A major application of covering arrays (CAs) arises in software interaction testing [1], where a covering array can be used to represent an interaction test suite as follows. In a software test, we have 𝑘 components or factors. Each of these parameters has 𝑣 values or levels. A test suite is an 𝑁×𝑘 array where each row is a test case. Each column represents a component, and a value in the column is the particular configuration. By mapping a software test problem to a covering array of strength 𝑡, we can guarantee that we have tested, at least once, all 𝑡-way combinations of component values [2]. Thus, software testing costs can be substantially reduced by minimizing the number of test cases 𝑁 in the covering array. Please observe that software interaction testing is a black-box testing technique, and thus it exhibits weaknesses that should be addressed by employing white-box testing techniques. For a detailed example of the use of covering arrays in software interaction testing, the reader is referred to [3].

In this paper, we aim to develop an enhanced sequential simulated annealing (ESSA) algorithm for finding near-optimal covering arrays. Simulated annealing algorithm is a general-purpose stochastic optimization technique that has proved to be an effective tool for approximating globally optimal solutions to many optimization problems. However, one of the major drawbacks of the technique is the time it requires to obtain good solutions (moreover, when the evaluation function requires too much time). To address this drawback, we propose three parallel simulated annealing approaches to solve the CAC problem. The objective is to find the best bounds to some ternary covering arrays by using parallelism. To our knowledge, the application of parallel simulated annealing to the CAC problem has not been reported in the literature. Some methods of parallelization of simulated annealing are discussed in [4–8].

The remainder of this paper is organized as follows. Some techniques that have been used for constructing covering arrays are presented in Section 2. Section 3 describes the components of our sequential annealing algorithm. In Section 4, three parallel simulated annealing approaches are discussed. Section 5 describes the experimental results. Finally, Section 6 presents the conclusions derived from the research presented in this paper.

2. Review of Covering Arrays Construction Methods

Because of the importance of the construction of (near) optimal covering arrays, much research has been carried out in developing effective methods for constructing them. There are several reported methods for constructing these combinatorial models. Among them are (1) direct methods; (2) recursive methods; (3) greedy methods; (4) metaheuristics methods.

Chateauneuf and Kreher [9] introduced a new method to construct covering arrays of strength three. This construction uses the structure of covering arrays and the repetition in covering arrays. The idea is to construct a covering array from a small array, a starter vector, and a group. This construction builds the covering array column by column by considering the group acting on the columns of the starter vector. Meagher and Stevens [10] extended the idea exhibited in [9], presenting a strategy for obtaining the starter vector by local search and the selection of a group action. Recently, Lobb et al. [11] presented a generalization of this method to permit any number of fixed points, permit an arbitrary group acting on the symbols, and permit an arbitrary group acting on the columns. With all these generalizations were obtained new bounds for covering arrays of strength two.

Tang and Woo [12] used constant weight vectors to construct test suites to be applied to logic circuit testing. Martinez-Pena et al. [13] introduced a new method for constructing covering arrays using trinomial coefficients, and they improved the results presented in [12]. Martinez-Pena et al. used the trinomial coefficients for the representation of the search space in the construction of ternary covering arrays. It is clear that any covering array is formed by a row set. In this sense, a trinomial coefficient represents a particular subset of rows which may belong to a ternary covering array.

Hartman [3] presented a recursive construction which gives a method of squaring the number 𝑘 of columns in a covering array of strength 𝑡 while multiplying the rows 𝑁 by a factor dependent only on 𝑡 and 𝑣, but independent of 𝑘. This factor is related to the Turan numbers 𝑇(𝑡;𝑣) that are defined to be the number of edges in the Turan graph.

Colbourn et al. [14] presented a product construction for 𝑡=2. In general, the product of two covering arrays where 𝑡=2 consists in obtaining a new covering array where the number of columns is equal to the product of the columns of the ingredients, and the number of rows is equal to the sum of the rows of each ingredient.

The majority of commercial and open-source test data generating tools use greedy algorithms for covering arrays construction (AETG, TCG, ACTS, DDA, among others).

Cohen et al. [1] presented a strategy called AETG. In AETG, covering arrays are constructed one row at a time. To generate a row, the first 𝑡-tuple is selected based on the one involved in most uncovered pairs. Remaining factors are assigned levels in a random order. Levels are selected based on the one that covers the most new 𝑡-tuples. For each row that is actually added to the covering array, there are a number of, 𝑀, candidate rows that are generated, and only a candidate that covers the most new 𝑡-tuples is added to the covering array. Once a covering is constructed, a number, 𝑅, of test suites are generated and the smallest test suite generated is reported. This process continues until all pairs are covered.

Tung and Aldiwan [15] proposed a tool called TCG. In TCG, one row is added at a time to a covering array until all pairs are covered. Before each row is added, a number of up to 𝑀 candidate rows are generated, and the best candidate (covering the most new pairs) is added. 𝑀 is defined to be the maximum cardinality of factors (the maximum number of levels associated with any factor). To construct each row, factors are assigned levels in an order based on a nonascending order of the cardinality of each factor. Each level for the factor is evaluated, and a count of the number of pairs that are covered is used to determine whether or not to select a level for a factor.

Bryce and Colbourn [16] presented an algorithm called DDA. The DDA constructs one row of a covering array at a time using a steepest ascent approach. Factors are dynamically fixed one at a time in an order based on density. New rows are continually added until all interactions have been covered.

Lei and Tai [17] introduced a new algorithm called IPO, for pairwise testing. For a system with two or more input parameters, the IPO strategy generates a pairwise test set for the first two parameters, extends the test set to generate a pairwise test set for the first three parameters, and continues to do so for each additional parameter. Contrary to many other algorithms that build covering arrays one row at a time, the IPO strategy constructs them one column at a time. Lei et al. [18] introduced an algorithm for the efficient production of covering arrays, called IPOG, which generalizes the IPO strategy from pairwise testing to multiway testing. The main idea is that covering arrays of 𝑘−1 columns can be used to efficiently build a covering array with degree 𝑘.

Ronneseth and Colbourn [19] introduced a new algorithm for constructing covering arrays, the BBA. The BBA's fundamental idea is to combine smaller covering arrays by reordering the rows and then to append additional rows for the remaining uncovered pairs.

Some stochastic algorithms in artificial intelligence, such as tabu search [20, 21], simulated annealing [22], generic algorithms, and ant colony optimization algorithm [23], provide an effective way to find approximated solutions. In these algorithms, the optimization focuses on one value of 𝑁 at a time, attempting to find a covering array for that size.

A simulated annealing metaheuristic (henceforth called SAC) has been applied by Cohen et al. in [22] for constructing covering arrays. SAC starts with a randomly generated initial solution 𝑀 whose cost 𝑐(𝑀) is measured as the number of uncovered 𝑡-tuples. In their implementation, Cohen et al. use a simple geometric function 𝑇_𝑛=0.9998𝑇_𝑛−1 with an initial temperature fixed at 𝑇_𝑖=0.20. At each temperature, 2000 neighboring solutions are generated. The algorithm stops either if a valid covering array is found, or if no change in the cost of the current solution is observed after 500 trials. The authors justify their choice of these parameter values based on some experimental tuning. They conclude that their simulated annealing implementation is able to produce smaller covering arrays than other computational methods, sometimes improving upon algebraic constructions.

Torres-Jimenez and Rodriguez-Tello [24] introduced a new simulated annealing implementation for constructing binary covering arrays.

3. Sequential Simulated Annealing

In this section, we briefly review simulated annealing (SA) algorithm and propose an enhanced sequential simulated annealing (ESSA) implementation to solve the CAC problem. ESSA is an extension of the simulated annealing presented in [24] for constructing covering arrays for 𝑣>2 and mixed-level covering arrays.

Simulated annealing is a randomized local search method based on the simulation of annealing of metal. The acceptance probability of a trial solution is given by (3.1), where 𝑇 is the temperature of the system, Δ𝒞 is the difference of the costs between the trial and the current solutions (the cost change due to the perturbation), and (3.1) means that the trial solution is accepted by nonzero probability 𝑒^{−Δ𝒞/𝑇} even though the solution deteriorates (uphill move) (ℙ)=𝑒1ifΔ𝒞<0,_{−Δ𝒞/𝑇}otherwise.(3.1)

Uphill moves enable the system to escape from the local minima; without them, the system would be trapped into a local minimum. Too high of a probability for the occurrence of uphill moves, however, prevents the system from converging. In simulated annealing, the probability is controlled by temperature in such a manner that at the beginning of the procedure the temperature is sufficiently high, in which a high probability is available, and as the calculation proceeds, the temperature is gradually decreased, lowering the probability [25].

The following paragraphs will describe each of the components of the implementation of our simulated annealing implementation. The description is done given the matrix representation of a covering array.

3.1. Internal Representation

Let 𝑀 be a potential solution in the search space ℳ, that is, a covering array CA(𝑁;𝑡,𝑘,𝑣) of size 𝑁, strength 𝑡, degree 𝑘, and order 𝑣. Then 𝑀 is represented as an 𝑁×𝑘 array on 𝑣 symbols, in which the element 𝑚_𝑖,𝑗 denotes the symbol assigned in the test configuration 𝑖 to the parameter 𝑗. The size of the search space ℳ is then given by ||ℳ||=𝑣^𝑁𝑘.(3.2)

3.2. Initial Solution

The initial solution 𝑀 is constructed by generating 𝑀 as a matrix with maximum Hamming distance. The Hamming distance 𝑑(𝑥,𝑦) between two rows 𝑥,𝑦∈𝑀 is the number of elements in which they differ. Let 𝑟_𝑖 be a row of the matrix 𝑀. To generate a random matrix 𝑀 of maximum Hamming distance, the following steps are performed: (1)generate the first row 𝑟₁ at random,(2)generate 𝑠 rows 𝑐₁,𝑐₂,…,𝑐_𝑠 at random, which will be candidate rows, (3)select the candidate row 𝑐_𝑖 that maximizes the Hamming distance according to (3.3) and added to the 𝑖th row of the matrix 𝑀, (4)repeat from step 2 until 𝑀 is completed; 𝑔𝑟_𝑖=^𝑖−1_𝑘𝑠=1_𝑣=1𝑑𝑚_𝑠,𝑣,𝑚_𝑖,𝑣,where𝑑𝑚_𝑠,𝑣,𝑚_𝑖,𝑣=1if𝑚^𝑠,𝑣0otherwi≠se𝑚._𝑖,𝑣,(3.3)

An example is shown in (3.4); the number of symbols different between rows 𝑟₁ and 𝑐₁ is 4 and between 𝑟₂ and 𝑐₁ is 3 summing up 7. Then, the Hamming distance for the candidate row 𝑐₁ is 7: Rows⎧⎪⎨⎪⎩𝑟¹𝑟=2101₂𝑐=1201₁=0210,Distances⎧⎪⎨⎪⎩𝑑𝑟¹𝑑𝑟,𝑐₁=4₂𝑔𝑐,𝑐₁=3₁=7,(3.4)

Equation (3.4) is an example of the Hamming distance between two rows 𝑟₁, 𝑟₂ that are already in the matrix 𝑀 and a candidate row 𝑐₁.

3.3. Evaluations Function

The evaluation function 𝒞(𝑀) is used to estimate the goodness of a candidate solution. Previously reported metaheuristic algorithms for constructing covering arrays have commonly evaluated the quality of a potential solution (covering array) as the number of combination of symbols missing in the matrix 𝑀 [20, 22, 23]. Then, the expected solution will be zero missing. In the proposed simulated annealing implementation, this evaluation function definition was used. Its computational complexity is equivalent to 𝑂(𝑁^𝑘𝑡).

3.4. Neighborhood Function

Given that ESSA is based on local search (LS), then a neighborhood function must be defined. The main objective of the neighborhood function is to identify the set of potential solutions which can be reached from the current solution in a local search (LS) algorithm. In case two or more neighborhoods exhibit complementary characteristics, it is then possible and interesting to create more powerful compound neighborhoods. The advantage of such an approach is well documented in [26]. Following this idea, and based on the results of our preliminary experimentations, a neighborhood structure composed by two different functions is proposed for ESSA.

The neighborhood function 𝒩₁(𝑠) makes a random search of a missing 𝑡-tuple and then tries by setting the 𝑗th combination of symbols in every row of 𝑀. The neighborhood function 𝒩₂(𝑠) randomly chooses a position (𝑖,𝑗) of the matrix 𝑀 and makes all possible changes of symbol. During the search process, a combination of both 𝒩₁(𝑠) and 𝒩₂(𝑠) neighborhood functions is employed by ESSA algorithm. The former is applied with probability ℙ, while the latter is employed at a (1−ℙ) rate. This combined neighborhood function 𝒩₃(𝑠,𝑥) is defined in (3.5), where 𝑥 is a random number in the interval [0,1]: 𝒩₃(𝑠,𝑥)=𝒩¹𝒩(𝑠)if𝑥≤ℙ,₂(𝑠)if𝑥>ℙ.(3.5)

3.5. Cooling Schedule

The cooling schedule determines the degree of uphill movement permitted during the search and is thus critical to the ESSA algorithm's performance. The parameters that define a cooling schedule are an initial temperature, a final temperature or a stopping criterion, the maximum number of neighboring solutions that can be generated at each temperature, and a rule for decrementing the temperature. The literature offers a number of different cooling schedules, see, for instance, [4, 27]. ESSA uses a geometrical cooling scheme mainly for its simplicity. It starts at an initial temperature 𝑇_𝑖 which is decremented at each round by a factor 𝛼 using the relation 𝑇_𝑘=𝛼𝑇_𝑘−1. For each temperature, the maximum number of visited neighboring solutions is 𝐿. It depends directly on the parameters (𝑁, 𝑘, and 𝑣) of the studied covering array. This is because more moves are required for covering arrays with alphabets of greater cardinality.

3.6. Termination Condition

The stop criterion for ESSA is either when the current temperature reaches 𝑇_𝑓, when it ceases to make progress, or when a valid covering array is found. In the proposed implementation, a lack of progress exists if after 𝜙 (frozen factor) consecutive temperature decrements the best-so-far solution is not improved.

3.7. Simulated Annealing Pseudocode

Algorithm 1 presents the simulated annealing heuristic as described above. The meaning of the three functions is obvious: INITIALIZE computes a start solution and initial values of the parameters 𝑇 and 𝐿; GENERATE selects a solution from the neighborhood of the current solution, using the neighborhood function 𝒩₃(𝑠,𝑥); CALCULATE_CONTROL computes a new value for the parameter 𝑇 (cooling schedule) and the number of consecutive temperature decrements with no improvement in the solution.

(1)  INITIALIZE(𝑀,𝑇,𝐿)  /* Create the 𝑖𝑛𝑖𝑡𝑖𝑎𝑙𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛. */ (2)  𝑀⋆←𝑀;   /* Memorize the 𝑏𝑒𝑠𝑡𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛. */ (3)  𝑇=𝑇0;   /* Initial 𝑡𝑒𝑚𝑝𝑒𝑟𝑎𝑡𝑢𝑟𝑒 of SA. */ (4)  repeat (5)  for  𝑖←1  to  𝐿  do (6)   𝑀𝑖←GENERATE(𝑀);    /* Perturb current state. */ (7)  Δ𝒞←𝒞(𝑀𝑖)−𝒞(𝑀);    /* Evaluate cost function. */ (8)  𝑥←random;     /*𝑥 in the range [0,1). */ (9)  if  Δ𝒞<0𝐨𝐫𝑒−Δ𝒞/𝑇>𝑥  then (10)   𝑀←𝑀𝑖;     /* Accept new state.*/ (11)   if  𝒞(𝑀)<𝒞(𝑀⋆)   then (12)    𝑀⋆←𝑀; /* Memorize the 𝑏𝑒𝑠𝑡𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛. */ (13)   end if (14)  end if (15) end for (16) CALCULATE_CONTROL(𝑇,𝜙) (17)  until  𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑡𝑖𝑜𝑛𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 is satisfied;

Algorithm 1 

Sequential simulated annealing for the CAC problem.

This is unlike the classical method which takes as a solution to the problem the last value obtained in the annealing chain [28]. We memorize the best solution found during the whole annealing process (see lines 2 and 12).

In the next section, it is presented three parallel simulated annealing approaches for solving the CAC problem.

4. Parallel Simulated Annealing

Parallelization is recognized like a powerful strategy to increase algorithms efficiency; however, simulated annealing parallelization is a hard task because it is essentially a sequential process.

In evaluating performance of a parallel simulated annealing (PSA), it needs to consider solution quality as well as execution speed. The execution speed may be quantified in terms of speed-up (𝒮) and efficiency (ℰ). The 𝒮 is defined as the ratio of the execution time (on one processor) by the sequential simulated annealing to that by the PSA (on 𝑃 processors) for an equivalent solution quality. In the ideal case, 𝒮 would be equal to 𝑃. The ℰ is defined as the ratio of the actual 𝒮 to the ideal 𝒮(𝑃).

Next, we propose three parallel implementations of the simulated annealing algorithm described in Section 3. For these cases, let 𝑃 denote the number of processors and 𝐿 the length of Markov chain.

4.1. Independent Search Approach

A common approach to parallelizing simulated annealing is the ISA [4, 6, 29]. In this approach, each processor independently perturbs the configuration, evaluates the cost, and decides on the perturbation. The processors 𝑃_𝑖,𝑖=0,1,…,𝑃−1 carry out the independent annealing searches using the same initial solution and cooling schedule as in the sequential algorithm. At each temperature, 𝑃_𝑖 executes 𝑁×𝑘×𝑣² annealing steps. When each processor finishes, it sends its results to processor 𝑃₀. Finally, processor 𝑃₀ chooses the final solution among the local solutions.

We have implemented a simulated annealing algorithm using ISA approach for constructing covering arrays. In the developed implementation, the processors do not interact during individual annealing processes until all processors find their final solution. Then, the best of the solutions is saved and the others are discarded.

4.2. Semi-Independent Search Approach

Aarts and van Laarhoven [4] introduced a new parallel simulated annealing algorithm named division algorithm. In the division algorithm, the number of iterations at each temperature is divided equally between the processors. After a change in temperature, each processor may simply start from the final solution obtained by that processor at the previous temperature. The best solution from all the processors is then taken to be the final solution. Another variant of this approach is to communicate the best solution from all the processors to each processor every time the temperature changes. Aarts and van Laarhoven [4] found no significant differences in the performance of these two variants.

We have developed an implementation of division algorithm; we named the implementation SSA. In SSA, parallelism is obtained by dividing the effort of generation a Markov chain over the available processors. A Markov chain is divided into 𝑃 subchains of the length ⌊𝐿/𝑃⌋. In this approach, the processors exchange local information including intermediate solutions and their costs. Then, each processor restarts from the best intermediate ones.

Compared to the ISA, communication overhead in this SSA approach would be increased. However, each processor can utilize the information from other processors such that the decrease in computations and idle times can be greater than the increase in communication overhead. For instance, a certain processor which is trapped in an inferior solution can recognize its state by comparing it with others and may accelerate the annealing procedure. That is, processors may collectively converge to a better solution.

4.3. Cooperative Search Approach

In order to improve the performance of the SSA approach, we propose the cooperative search approach (CSA); it used asynchronous communication among processors accessing the global state to eliminate the idle times. Each processor follows a separate search path, accesses the global state which consists of the current best solution and its cost whenever it finished a Markov subchain, and updates the state if necessary. Once a processor gets the global state, it proceeds to the next Markov subchain with any delay.

Unlike SSA, CSA has the following characteristics:(i)idle times can be reduced since asynchronous communications overlap a part of the computation; (ii)less communication overhead, an isolated access to the global state is needed by each processor at the end of each Markov subchain; (iii)the probability of being trapped in a local optimum can be smaller. This is because not all the processors start from the same state in each Markov subchain.

5. Experimental Results

This section presents an experimental design and results derived from testing the parallel ISA, SSA, and CSA algorithms described in the Section 4. In order to show the performance of these approaches, three experiments were developed. The first experiment had as purpose to fine-tune the probabilities of the neighborhood functions to be selected. The second experiment had as purpose to compare the three approaches in terms of parallel execution time. Among the three approaches, the CSA approach was the fastest. The third experiment evaluated the quality of the solutions of the CSA approach over a new benchmark proposed in this paper. The results were compared against the best-known solutions reported in the literature to construct covering arrays [30].

The three parallel approaches were implemented using C language and the message passing interface (MPI) library. The implementations were run on the Tirant supercomputer (The Tirant supercomputer: http://www.uv.es/siuv/cas/zcalculo/res/informa.wiki). Tirant comprises 256 JS20 compute nodes (blades) and 5 p515 servers. Every blade has two IBM Power4 processors at 2.0 GHz running Linux operating system with 4 GB of memory RAM and 36 GB of local disk storage. All the servers provide a total of nearly 10 TB of disk storage accessible from every blade through GPFS (global parallel file system). Tirant has in total 512 processors, 1 TB of memory RAM, and 19 TB of disk storage. The following parameters were used for all simulated annealing implementations:(1)Initial temperature 𝑇_𝑖=4.0,(2)Final temperature 𝑇_𝑓=1.0𝐸−10,(3)Cooling factor 𝛼=0.99,(4)Maximum neighboring solutions per temperature 𝐿=𝑁×𝑘×𝑣²,(5)Frozen factor 𝜙=11;(6)The neighborhood function 𝒩₃(𝑠,𝑥) is applied using a probability ℙ=0.3.

5.1. Tuning of the Parameters of Parallel Simulated Annealing

It is well known that the performance of a simulated annealing algorithm is sensitive to parameter tuning. In this sense, we follow a methodology for a fine-tuning of the two neighborhood functions used in ESSA. The fine-tuning was based on the linear Diophantine equation (5.1), where 𝑥_𝑖 represents a neighborhood function, and its value set to 1, ℙ_𝑖 is a value in {0.0,0.1,…,1.0} that represents the probability of executing 𝑥_𝑖, and 𝑞 is set to 1.0 which is the maximum probability of executing any 𝑥_𝑖: ℙ₁𝑥₁+ℙ₂𝑥₂=𝑞.(5.1)

A solution to the given linear Diophantine equation must satisfy (5.2). This equation has 11 solutions, each solution is an experiment that tests the degree of participation of each neighborhood function in ESSA to accomplish the construction of a covering array. Every combination of the probabilities was applied by ESSA to construct the set of covering arrays shows in Table I(a), and each experiment was run 31 times; with the data obtained for each experiment, we calculate the median. A summary of the performance of ESSA with the probabilities that solved the 100% of the runs is shown in Table I(b): ²_𝑖=1ℙ_𝑖𝑥_𝑖=1.0(5.2)

Table 1 

(a) A set of 7 covering arrays configurations. (b) Performance of ESSA. Columns 3–9 show the time taken to construct each of the covering arrays.

Finally, given the results shown in Figure 1, the best configuration of probabilities was ℙ₁=0.3 and ℙ₂=0.7 because it found the covering arrays in smaller time (median value). The values ℙ₁=0.3 and ℙ₂=0.7 were kept fixed in the second experiment.

Figure 1 

Performance of ESSA with the 11 combinations of probabilities.

5.2. Comparison of the ISA, SSA, and CSA Approaches

To test the performance of the ISA, SSA, and CSA approaches, we propose the construction of a covering array with 𝑁=80, 𝑡=3, 𝑘=22, and 𝑣=3. Each approach was executed 31 times (for providing statistical validity to experiment) using 𝑃={4,8,16,32}.

The performance of the algorithms has been compared based on median speedup as a function of the number of processors; the results are shown in Figure 2.

Figure 2 

Median speedup for the ISA, SSA, and CSA approaches.

The ISA approach; had difficulty in handling the large problem instances, it does not scale well. The SSA approach provides reasonable results; however, because it is a synchronous algorithm, the idle and communication times are inevitable. The CSA approach is the one which offers the best results; it reduces the execution time of the SSA approach by employing asynchronous information exchange.

In the next subsection, it is presented the third experiment of this work, and the purpose is to measure the performance of the CSA algorithm against the best-known solutions reported in the literature.

5.3. Comparison with State-of-the-Art Procedures

The purpose of this experiment is to carry out a performance comparison of the bounds achieved by the CSA approach with respect to the best-known solutions reported in the literature [30], which were produced using the following state-of-the-art procedures: orthogonal array construction, Roux type constructions, doubling constructions, algebraic constructions, deterministic density algorithm (DDA), Tabu search, and IPOG-F.

For this experiment, we have fixed the maximum computational time expended by our PSA for constructing a covering array to 72 hours and 50 processors. We create a new benchmark composed of 182 covering arrays distributed as follows:(i)47 covering arrays with strength 𝑡=3, degree 4≤𝑘<50, and order 𝑣=3,(ii)46 covering arrays with strength 𝑡=4, degree 5≤𝑘<50, and order 𝑣=3,(iii)45 covering arrays with strength 𝑡=5, degree 6≤𝑘<50, and order 𝑣=3,(iv)44 covering arrays with strength 𝑡=6, degree 7≤𝑘<50, and order 𝑣=3,

The detailed results produced by this experiment are listed in Table 2. The first two columns in each subtable indicate the strength 𝑡 and the degree 𝑘 of the selected instances. Next two columns show, in terms of the size 𝑁 of the covering arrays, the best-known solution reported in the literature and the improved bounds produced by the CSA approach. Last column depicts the difference between the best result produced by CSA approach and the best-known solution (Δ=𝛽−𝜗).

Table 2 

It shows the improved bounds produced by CSA approach. Column 𝜗 represents the best-known solution reported in the literature [30]. Column 𝛽 represents the best solution in terms of 𝑁 produced by CSA approach. Last column (Δ) depicts the difference between the best result produced by CSA approach and the best-known solution (Δ=𝛽−𝜗). (a) Improved bounds on CAN(3,𝑘,3); (b) improved bounds on CAN(4,𝑘,3); (c) improved bounds on CAN(5,𝑘,3); (d) improved bounds on CAN(6,𝑘,3).

Figure 3 compares the results shown in Table 2 involving the CSA algorithm and the best-known solutions. The analysis of the data presented led us to the following observation. The solutions quality attained by the CSA approach is very competitive with respect to that produced by the state-of-the-art procedures summarized in column 3 (𝜗). In fact, it is able to improve on 134 previous best-known solutions.

(a) CAN(3,𝑘,3)

(b) CAN(4,𝑘,3)

(c) CAN(5,𝑘,3)

(d) CAN(6,𝑘,3)

Figure 3 

Graphical comparison of the quality solutions between CSA and the state of the art [30].

6. Conclusions

The long execution time of simulated annealing due to its sequential nature hinders its application to realistically sized problems, in this case, the CAC problem when 𝑡>3,5<𝑘≤100, and 𝑣=3. A more efficient way to reduce execution time and make the simulated annealing a more promising method is to parallelize sequential simulated annealing. It is a challenging task. In fact, there are many approaches that may be considered in parallelizing simulated annealing. However, an inappropriate strategy used will likely result in poor performance.

In this paper, we have used three different approaches to do the parallelization of the simulated annealing algorithm. From the experimental results, we found that the ISA approach has the worst performance, and it does not scale well. The SSA approach offers reasonable execution times; compared to the ISA, communication overhead in the SSA approach would be increased when the size of the problem grows. However, each processor can utilize the information from other processors such that the decrease in computations and idle times can be greater than the increase in communication overhead. For instance, a certain processor which is trapped in an inferior solution can recognize its state by comparing it with others and may accelerate the annealing procedure. That is, processors may collectively converge to a better solution.

The CSA approach is the one which offers the best results; it significantly reduces the execution time of the SSA approach by employing asynchronous information exchange. The quality solutions attained by the CSA approach are very competitive with respect to that produced by the state-of-the-art procedures; in fact, it is able to improve on 134 previous best-known solutions and equals the solutions for other 29 instances.

These experimental results confirm the practical advantages of using CSA algorithm in the software testing area. It is a robust algorithm yielding smaller test suites than other representative state-of-the-art algorithms, which allows reducing software testing costs.

The new bounds are available in Cinvestav covering array repository (CAR), which is available under request at http://www.tamps.cinvestav.mx/~jtj/authentication.php. We have verified all covering arrays described in this paper using the tool described in [31].

Acknowledgments

The authors thankfully acknowledge the computer resources and assistance provided by Spanish Supercomputing Network (TIRANT-UV). This research work was partially funded by the following projects: CONACyT 58554; Cálculo de Covering Arrays; 51623-Fondo Mixto CONACyT; Gobierno del Estado de Tamaulipas.