Complexity

Complexity / 2021 / Article
Special Issue

Artificial Intelligence for Smart System Simulation

View this Special Issue

Research Article | Open Access

Volume 2021 |Article ID 6660572 | https://doi.org/10.1155/2021/6660572

Lucero Ortiz-Aguilar, Martín Carpio, Alfonso Rojas-Domínguez, Manuel Ornelas-Rodriguez, H. J. Puga-Soberanes, Jorge A. Soria-Alcaraz, "A Methodology to Determine the Subset of Heuristics for Hyperheuristics through Metalearning for Solving Graph Coloring and Capacitated Vehicle Routing Problems", Complexity, vol. 2021, Article ID 6660572, 22 pages, 2021. https://doi.org/10.1155/2021/6660572

A Methodology to Determine the Subset of Heuristics for Hyperheuristics through Metalearning for Solving Graph Coloring and Capacitated Vehicle Routing Problems

Academic Editor: Abd E.I.-Baset Hassanien
Received06 Oct 2020
Revised15 Mar 2021
Accepted12 Apr 2021
Published26 Apr 2021

Abstract

In this work, we focus on the problem of selecting low-level heuristics in a hyperheuristic approach with offline learning, for the solution of instances of different problem domains. The objective is to improve the performance of the offline hyperheuristic approach, identifying equivalence classes in a set of instances of different problems and selecting the best performing heuristics in each of them. A methodology is proposed as the first step of a set of instances of all problems, and the generic characteristics of each instance and the performance of the heuristics in each one of them are considered to define the vectors of characteristics and make a grouping of classes. Metalearning with statistical tests is used to select the heuristics for each class. Finally, we used the Naive Bayes to test the set instances with k-fold cross-validation, and we compared all results statistically with the best-known values. In this research, the methodology was tested by applying it to the problems of capacitated vehicle routing (CVRP) and graph coloring (GCP). The experimental results show that the proposed methodology can improve the performance of the offline hyperheuristic approach, correctly identifying the classes of instances and applying the appropriate heuristics in each case. This is based on the statistical comparison of the results obtained with those of the state of the art of each instance.

1. Introduction

In Computer Science, a heuristic is a technique designed to solve a problem when classical methods fail to find an exact solution or when they are too slow. Currently, there is great interest from the scientific community in offering ad hoc heuristic solutions for real-world optimization problems. To achieve this, it is necessary to have a priori knowledge of the problem and, often, computationally efficient solutions are produced in a reasonable time. However, the no free lunch theorem mentions [1] that no methodology or algorithm can solve all problems; that is, ad hoc heuristics are usually not generalizable, and they do not always work well when applied to other problems even when they share some similar characteristics. This fact has led research efforts towards the development of general-purpose search methodologies known as hyperheuristics, whose main characteristic is that they are independent of the problem domain.

Hyperheuristics can be classified according to their learning methods, such as no learning, online learning [2], and offline learning [3]. In the context of combinatorial optimization, hyperheuristics are defined as “heuristics to choose heuristics” [4], or as “an automated methodology for selecting or generation heuristics to solve computational search problems,” [4]. According to Pillay in [2], the generality of a hyperheuristic can be seen from three levels: a generalization on instances of problems, generalization for a particular problem, and a generalization focused on different types of problems, the latter being high level. Some variations of hyperheuristics depend on the type of learning used (e.g., with online learning [2] and offline learning [3]) or the nature of the heuristics.

One of the main problems in hyperheuristics is to propose methodologies that allow generating and/or selecting the minimum set of heuristics that perform well for the problem at hand and this heuristic set is usually selected by expert researchers in the field [2]. In order to automatically select the best heuristic that performs best for the problem, an approach called metalearning was proposed [5, 6] and its use in hyperheuristics can be found in Amaya et al. [7]. Likewise, the different meanings used and taxonomy for each interaction between metalearning and optimization were studied by Song et al. [8].

As there is no methodology or algorithm that can solve all problems, our objective is to base ourselves on information about the problem and the performance of the algorithms to provide this knowledge to hyperheuristics. Metalearning generates metaknowledge, and we use this to select better heuristics for solving problems. With this approach, we pretend to propose methodologies that are like the intelligence of humans. Humans could learn from problems, their characteristics, their variables, and their restrictions, and after elaborating an analysis or discernment, the “human expert” proposes the best tool and solves the problem.

In this paper, we propose a methodology to determine a subset of heuristics for hyperheuristics through metalearning and partition for solving different problems (described below) without ad hoc adjustments by providing information about the problem and the performance of the heuristics to the hyperheuristic. It is well known that the correct characterization is the key to selecting the best heuristic [6]. Consequently, this affects the hyperheuristic design, and that is why in our approach we decided to use offline learning. Metalearning consists of two basic parts, the metafeatures and the metalearner; the first are generated from the information of the problem and the solution algorithms, while the second uses a grouping technique. Our methodology extends beyond the classic metalearner approach, and we apply nonparametric statistical tests to determine which heuristics will provide the same performance if the full set of heuristics is applied.

In order to test our proposal, we used two different well-known problem domains: capacitated vehicle routing problem and graph coloring problem. The capacitated vehicle routing problem (CVRP) has different restrictions such as minimizing distance, time, capacity, and delivery. This problem aims to find the subtour of n cities, without repeating two cities on the same tour or different tours. In the CVRP state of the art, there exist different variants that consider this basic definition and extra restrictions. On the other hand, the graph coloring problem consists of labeling each vertex of a given graph with k-colors and it is a well-known problem, which has been solved by exact methods, heuristics, metaheuristics, and hyperheuristics. Although each problem can be solved by ad hoc heuristics, to date, there is no general methodology capable of solving all variants of both problems. The use of partitions for constraint satisfaction problems, such as university timetabling and VRP, has had good results [9, 10]. Although there are taxonomies of the said problems, characterizing and classifying the instances under a hyperheuristic context with metalearning with a statistical test is an approach that has not been explored in the literature.

Finally, it is worth mentioning that to do a better design or choice of hyperheuristics and predict which is the best algorithm according to the classification of the previous instance, we propose to use metalearning with statistical analysis of the heuristics, which will allow improving these points. Our proposal also provides information that allows us to understand the performance of heuristics and hyperheuristics in the problem of interest.

The remaining content of this article contains a description of related work in Section 2, which covers a review on heuristics, hyperheuristics, and metalearning. Problem definitions and theories related to heuristics are reported in Section 3. Sections 4 and 5 present the proposed methodology. The found results and findings including performance comparison are described in Section 6. Finally, concluding remarks are presented in Section 7.

In this section, we will give a view of the heuristic and hyperheuristic algorithms. Moreover, we will define some basic concepts of metalearning. Finally, we will give a discussion of the pros and cons of the presented methods.

2.1. Heuristics

We made an extensive review of different heuristics applied to CVRP and GCP. We selected a total of 11 heuristics that, after a previous experimental analysis, those heuristics apply to both problems and we list them below.

K-flip or K-opt heuristic was proposed by Lin and Kernighan in [11] for the travel salesman problem (TSP). This heuristic was based on the general interchange transformation, i.e., a city must change its position with another city on the same tour. Besides, this heuristic is one of the most popular for TSP [12]; it has been applied in other problems such as planar graphs, unconstrained binary quadratic programming, and the study of its complexity in SAT and MAX-SAT. The two-point perturbation is a case of k-flip, and we give a detailed description and algorithm of these heuristics in the following sections.

The k-swap heuristic is similar and frequently confused with k-flip. The K-swap heuristic improves its performance as a perturbation move when it uses two or three movements [13].

The move to less conflict heuristic, also known as minimizing conflicts, was proposed by Minton et al. [14]. The minimizing conflict heuristic has been applied to different areas in Computer Science such as hyperheuristics, graph coloring problems, pickup-and-delivery problems, and scheduling problems. The move to less conflict heuristic is a variant of the first fit, and the only difference is that the first one takes a random variable and changes its value for another that generates the least cost.

The first-fit heuristic was proved by Baker [15] for the bin-packing problem. On the other hand, in recent decades, this heuristic was applied to best-known problems such as bin packing, virtual machine relocation problem, and cutting stock. A remarkable variety of heuristics is worst fit which was studied by Baker [15] and Csirik [16], in particular, its application to bin-packing problem.

Soria-Alcaraz et al. [17] proposed three heuristics for university course timetabling best single perturbation (BSP), static-dynamic perturbation (SDP), and double dynamic perturbation as part of the pool low-level heuristics for hyperheuristic. Moreover, these heuristics were applied to the VRP in later research [18].

2.2. Hyperheuristics

We focused on offline learning hyperheuristics selection with perturbation heuristics, whose aim is to gather knowledge in the form of rules or programs, from training set instances. Usually, the offline selection hyperheuristics belong to machine learning methods, which are trained to create a tuned methodology for a problem domain [3]. Yates and Keedwell [19] demonstrated that subsequences of heuristics were found in the offline learning database that is effective for some problem domains. They used the Elman network to compute sequences of heuristics which were evaluated on unseen HyFlex example problems, and the results obtained are capable of intradomain learning and generalization with 99% confidence.

One of the crucial issues in hyperheuristics design is the quality and size of the heuristic pool [20]. Soria-Alcaraz et al. [20] proposed a methodology using nonparametric statistics and fitness landscape measurements for hyperheuristics design. This methodology was tested on course timetabling and vehicle routing problems; their hyperheuristic proposal had a compact heuristic pool and competed with some traditional methods in course timetabling. In the course timetabling problem, they obtained five best-known solutions of 24 PATAT instances [21]. Finally, a recent report by Amaya et al. [7] documented a model for creating selection hyperheuristics with constructive heuristics. The effectiveness of the model proposed by Amaya depends on the delta’s values used, which is useful with higher deltas.

2.3. Metalearning

The importance of metalearning, machine learning, and optimization has been studied by Song et al. [8]. The metalearning aim may concern accumulating and adapting experiences on the performance of multiple applications of a learning system. The metalearning field is also known as “learning to learn” [22] and it brings systems that can help by searching patterns across different tasks to control the process of exploiting cumulative expertise. The metalearning concept has been present in the field of heuristics and metaheuristics for TSP [23], the quadratic assignment problem, and hyperheuristics.

On the other hand, Gutierrez-Rodríguez et al. [23] used VRP with time windows and proposed a methodology based on metalearning to select the best metaheuristic for each instance. Besides, their proposal shared and exploited an offline scheme for the instant solutions of academies and industry. Their main contributions were to propose a set of features for characterizing VRPTW instances and design a classification process that predicts the most suitable metaheuristic for each instance. Nevertheless, they assumed that the solutions of the set instances could be stored, shared, and exploited in an offline scheme for predicting good solvers for new unseen instances.

The aim of this paper is not to present a survey on heuristics or hyperheuristics; our proposal is slightly different. Our proposal considers some vital aspects of the research, including the ones from Yates and Keedwell [19]. We took the offline hyperheuristic approach from Soria-Alcaraz et al. [20] and the statistical approach to selecting a pool heuristic from Kanda et al. [5]. The offline hyperheuristic approach is an effective and popular method in the machine learning area [8]. On the other hand, the statistical approach to selecting a pool heuristic is a useful and reliable method because it takes statistical information from the input data.

3. Combinatorial Problems

Our methodology is a general approach to competitive performance across several classes of problems. Thus, we used two problem domains: graph coloring and vehicle routing problems. In the following sections, we will review the formal definition of each of these problems as well as their benchmark instances.

3.1. Graph Coloring Problem

The graph coloring problem demonstration as an NP-hard problem was proposed by Karp [24]. According to [25], a formal vertex-coloring problem of a graph is a function , in which any two incident vertices are assigned different colors, that is, , and is a finite set of unordered pairs of vertices named edges, where the function is the coloring function and a graph for which there exists a vertex-coloring which requires colors is called k-colorable. The coloring function induces a partition of the graph into independent subsets where and . The benchmark instances can be found in http://mat.gsia.cmu.edu/COLOR/instances.htmlmat.gsia.cmu.edu. The above lets the partitioning methodology work on the input design and it is possible to avoid the ad hoc modifications to the heuristics since it will only pass a different objective function that adequately evaluates the instances of this problem.

3.2. Capacitated Vehicle Routing Problem

The capacitated vehicle routing problem (CVRP) is a variant of VRP [26]. In this problem, we have an undirected graph , vehicles, capacity, and a set of cities . Formally, the city is the and each vehicle must visit these cities starting from the and coming back to this. Alba and Dorronsoro [27] define a distance or travel time matrix between cities and . Each city has a demand of things . We denote it as a route , and is a permutation of the cities, starting and finishing at the depot . For each route, with . The cost of a problem solution is the sum of the costs of each route of aswhere k is the total of vehicles. This problem aims to determine for each vehicle the lowest cost (see equations (1) and (2)) tour or distance or travel time, considering the max capacity. Nota bene the hard constraints are the capacity of each vehicle and two vehicles cannot visit the same city. The CVRP has several constraints and a specific formal definition of this problem. These two characteristics let us apply our methodology with a design cities partition, where each vehicle is related to one part. As heuristics work with solutions that are already complete and respect important restrictions such as capacity, if any movement violates or exceeds this capacity, that solution is penalized in the objective function.

4. Methodology

For our methodology, it is important to know since the constraint modeling phase of the problem can be solved by partitions. The API-Carpio methodology and the methodology proposed by Soria-Alcaraz et al. [28] let us transform the instances of the problem with their restrictions, into inputs to apply the proposed methodology. The MMA matrix that is generated by applying the API-Carpio methodology lets us visualize the hard restrictions of the problem and evaluate the costs of visiting cities or nodes. For the soft constraints of the problem, the methodology proposed by Soria-Alcaraz et al. [28] is to be considered in the list of restrictions.

In this section, we describe the methodology to model the input data problem information used for the experimentation for two combinatorial problems based on the API-Carpio methodology. We integrated the API-Carpio methodology [29] and the methodology of the design proposed by Soria-Alcaraz et al. [28].

4.1. API-Carpio Methodology

This methodology is used to solve the university course timetabling problem and it considers three factors: students, teachers, and institutions (infrastructure). The methodology uses several structures for the equations previously described. One of the most important structures of this work is MMA. This matrix is constructed with information on the cities or nodes. For graph coloring, we use the information of the adjacency matrix, while for CVRP it is considered the cost matrix. Table 1 shows an example of an MMA matrix. The algorithm to construct this matrix is given in [29].


City/node

109997137
991018096
97803510
137961068

4.2. Methodology of Design

This methodology was extended from the proposal by Carpio [29] and their formal definition was proposed by Soria-Alcaraz et al. [28]. The methodology of design by Soria allows us to consider the objectives of course timetabling and to satisfy the different restrictions, by converting these to lists of time and space restrictions it is seeking to minimize student conflict.

To use this methodology, two structures are used to consider restrictions and variables: MMA matrix and LPH. The LPH has information about the possible restrictions that can be assigned to each node or city. An example of this list can be found in Table 2. The list shows in each row the number part, i.e., the node can be assigned in parts 1, 2, 4, 5, 7, or 8, but not in 6. The algorithm for generating artificial instances of LPH can be found in Ortiz-Aguilar [30].


Node or cityRestrictions

12345678
1234578
12345678
12345678

represents the node or cities.

5. Metalearning for Selecting a Subset of Heuristics for Hyperheuristics

According to Brazdil et al. [22], the approach of metalearning (ML) is to help the selection of an algorithm for a set of instances with metadata. According to Alpaydin [31], the metalearning aim is to find the best classifier for a set of data and to find the best classifier for the characterization when the data are considered. In our proposal, the data are associated with the problem instances of different problems, and the classifier is associated with the set of heuristics. Our objective is to be able to select the best set of heuristics for a hyperheuristic in a set of instances. In Figure 1, we show a diagram of the metalearning processes to obtain metaknowledge for the selection of heuristics (diagram modified from Brazdil and Giraud-Carrier [32] to our methodology).

In this work, we use metalearning to select a set of heuristics for hyperheuristics in a dataset. We named the set of characterized instances of the two problem domains as the “metacharacteristics” and the model that maps each instance to the corresponding group of heuristics for hyperheuristics the “metalearner.” In this case, the metalearner selected is the K-means algorithm. The methodology proposed in this article consists of 5 steps in the metalearning stage:(i)Step 1: obtain the set of instances to be worked on. In this case, we have as a criterion to select those instances that are susceptible to being resolved by partitions.(ii)Step 2: evaluation and extraction of characteristics of the instances. In this step, the characteristics of the heuristics and the instances are generated. Heuristics that apply to both problems are selected, this task becomes simple with the use of the partitioning methodology, and this is because it allows working always with generic inputs where the variables and restrictions are modeled. Later, heuristics work with these generic inputs and solutions that only have a fitness function corresponding to their problem (where the objectives are evaluated).(iii)Step 3: generation of metacharacteristics. Based on the characteristics of each problem and the performance of the heuristics applied to all instances, we generate vectors of characteristics that will be our metadata.(iv)Step 4: metalearning and the recommended model of heuristics. In the state of the art, research is limited to applying only a clustering technique for the recommendation of the algorithm model. We propose to incorporate a statistical analysis together with the clustering algorithm to improve the design of the basic subset of heuristics.

5.1. Problem Definition of Metalearning for Heuristic Selection

Consider a problem that belongs to the problem set (GCP). Let be a subset of low-level heuristics, which are used in the state of the art, to solve the problem . We denote as a random selection of heuristics , to be applied in the solution of , where , with to a reduced set of heuristics, that is, .

Let , where represents an empty string. We defined recursively where and . Then represents the set of all strings of length , formed from the symbols in . So Kleene’s closure from is [33]

When is omitted at the junction, we get the Kleene Plus closure :

In other words, is the collection of all possible nonempty strings of finite length generated from the symbols in .

Let be a heuristic selection hyperheuristic with offline training, where the training considers the set . After training, the provides a methodology with the best order of application of low-level heuristics, which we will denote by in the solution of the problem , which improves the performance of the application of a , where [17].

We take two problems and that belong to the problem set (GCP) that comply with level 3 of generality proposed by Pillay [2] and are susceptible to being solved by partitions. Let and be subsets of low-level heuristics, which are used in the state of art, to solve the problems and , and  = . We denote as a random selection of the heuristics , to be applied in the solution of and , where , with to a reduced set of heuristics, that is, .

Heuristic selection hyperheuristic is denoted as , with offline training, where the training considers the set. After training, the provides a methodology with the best order of application of low-level heuristics, which we will denote by in solving the problems which improve the performance of the application of a simple , where .

Our objective is to propose a methodology that provides the , with a reduced subset for its training, such that the provides a methodology , with the best order of application of the reduced set of heuristics , which we will denote by , in solving the problems and respectively, which equal the performance than with the application of the methodology where .

To solve the problem, the independent application of each of the heuristics of and was proposed, measuring their performance in solving the problems and . Apply statistical tests to contrast the performance of the independent heuristics and thereby discriminate from each set and those heuristics that obtained the lowest performance.

Next, we will focus on describing the stage of extracting characteristics from heuristics and instances. Our methodology improves the metalearning stage (step 4) with the application of nonparametric statistical tests to determine which heuristics are the ones that will provide the same performance if the full set of heuristics will be applied. This means that, if there are heuristics that are redundant, it is possible to leave them out and consider only those that enhance the speed of the search for solutions to the problem. The metalearning process proposed in this work to select the pool of heuristics includes the following steps:(1)The problems will be the source of information for the basic features.(2)Given a set of instances denoted as , for each of the instances apply a number of times the heuristic . The results will be the inner features. For the CVRP, a greedy heuristic is used, which will allow us to build feasible solutions to the problem. For graph coloring, we will initialize with a random construction heuristic. The next step is to apply the heuristics. It is possible after this step that the instances can be solved to the best solution due to their complexity. This means that it is possible to avoid executing a complete and expensive computation process when solving problems with the application of a simple heuristic.(3)With the information obtained from points 1 and 2, feature vectors will be formed that will be our metadata.(4)For a better treatment of the metadata from the previous step, the following steps are carried out:(a)The patterns that will be used in the k-means will be scaled with the following formula [34]:where are the values of the original variables (features).(b)Generate the pattern based on the inner and basic features per instance. The basic features will be the problem information and the inner features will be the fitness obtained by the heuristics applied times to the problem.(5)Pass the feature vectors to the clustering algorithm to form classes. According to Brazdil and Giraud-Carrier [32], the k-means is a simple learning method, which we apply to carry out the grouping of instances in classes. To determine which are the number of classes to form, we use Sturges rule, since, with data that are the potency of 2, it is approximated in a good way. Determine the subclasses with Sturges rule [35] withwhere T is the total amount of data.As the distance metric for the K-means, we use the Mahalanobis distance; this distance has properties such as being invariant to scale by nonsingular linear transformations. An in-depth study of different metrics [36, 37] will be a specific job to investigate whether it can improve the performance of the proposed methodology.(6)Label each pattern according to the group number in which each pattern (instance) was classified.(7)Apply again the three statistical tests to the results of heuristics per problem, according to the formulas in [38]. The test ranks 1 to the best performing heuristic, 2 to the second-best, and n to the worst-performing heuristic. From these tests, we will take the range of the heuristics and the range will now be considered as inner features. With this information and the class label, they will now form patterns.(8)Determine a cutoff point for each class based on the range, and in this case, it will be the average of the minimum and maximum range. Choose those heuristics that pass the cutoff criterion to be part of the minimum set.(9)The output will be the minimum set of heuristics per class.

This process is shown in a specific way in Figure 2. The two important aspects of metalearning in our work are heuristics and metacharacteristics, which are low-level heuristics and metafeatures.

5.2. Low-Level Heuristics

An important part of the hyperheuristic approach is the selection of the heuristic set. This article proposes to extract information from heuristics and problems to generate the metafeatures [32]. This lets us improve the design and testing of the hyperheuristic algorithm. The goal in this stage is to generate metafeatures in which the heuristics can have a better performance individually for all problem instances. This improves the next part in which the hyperheuristic must choose the sequence application for each heuristic and it uses a minimal pool of heuristics which is a fundamental part of it [2]. For all instances, it will be applied times for each heuristic. The heuristics were applied to the two problems and their respective instances were as follows:(1)K-Flip/Simple Random Perturbation (SRP) (). The heuristic changes the value of one or more variables (in some cases ) to another feasible value. The GCP aim is to change the color of a certain node to another [39]. Finally, for CVRP, the movement implies changing a city to another specific vehicle [12].(2)K-Swap/Kempe Chain Neighborhood/S-Chain Neighborhood (). It must be selected two or more varieties and then interchange their values among them when possible; otherwise, the change is not made. We exchange the color between nodes previously selected by GCP. This heuristic is using in works related to TSP or CVRP, also called k-interchange [40, 41].(3)Best Single Perturbation (BSP) (). This heuristic chooses a variable according to the list of hard restrictions (LPH) and changes its value. This exchange produces a better cost or in the worst case, the same cost [17]. Next time this heuristic is going to apply, the next variable will be chosen according to the next position of the last variable which was modified. The next node which must change color will be selected according to the last variable chosen for the graph coloring problem. The CVRP must be changing the city of the vehicle to another vehicle.(4)Static-Dynamic Perturbation (SDP) (). It is also known as statically dynamic perturbation (SDP). It is based on the variable selection with a probability distribution of the frequency in the last iterations. This heuristic chooses a variable and changes its value randomly [17]. The variables with fewer changes will have a higher probability to be selected. Applied to GCP, it would be a node with fewer color changes, and for the CVRP, the city has moved a few times to another vehicle.(5)Two Points Perturbation (2pp) (). It is also known as k-opt, and it is a particular case of the K-swap with a value of .(6)Double Dynamic Perturbation (DDP) (). This heuristic is based on the SDP, this receives a solution, and it modifies the value of a variable concerning a probability distribution. The difference is that a copy of the initial solution is kept and, in the end, the best of the two solutions is returned [17].(7)Move to Less Conflict (MLC) (). This selects a random variable, and it assigns to a part of the value which generates the least cost [18]. In GCP, the color changes according to another which improves the fitness, and in CVRP, the city is moved to another vehicle where the total distance of the route is minimized.(8)Min-Conflicts (). The heuristic selects a random variable, and it assigns to a part which generates the cheapest cost [18]. In GCP, the heuristic must change color from the selected node to another, which improves the result. For CVRP, the selected city with a lower cost in which it must minimize the total distance of the route.(9)First-Fit (). It changes the value of a variable to another, which is the least repeated in other variables [18], i.e., in CVRP, the heuristic will take a city and it will change it to the vehicle that has fewer cities in its route. For the GCP, it will select a node and it will assign the color that is least repeated.(10)Worse Fit (). It assigns the most repeated value if possible, without violating the hard constraints on a randomly selected variable [42]. For GCP and CVRP, we assign a node or city to the most repeated timeslot, color, or vehicle.(11)Burke–Abdullah (BA) (). This heuristic was proposed by Abdullah et al. [43], in which it chooses a variable applying Fail-First or Brelaz Heuristic [44] and its value changes according to the one that has obtained better performance by applying the following algorithms: minimum conflict, random selection, sequential selection, and least constrained.

5.3. Metafeatures

The description and generation of characteristics permit differentiation into at least two groups of instances within the same problem class. We used the terms of basic feature and based on the proposal conducted by Gutierrez-Rodríguez et al. [23]. As basic features, these are given by the problem, e.g., the number of nodes, colors, vehicles, and so on, depending on each problem information. For both classes of problems, the number of different is summarized in Table 3.


ProblemMin partitionMax partitionVariablesEdges

VRPVehiclesVehiclesCitiesCities connections
Graph coloringLower boundUpper boundNodesEdges

The fitness performance values of all heuristics are the inner feature key. Finally, the pattern per instance is basic feature+inner feature. The final pattern is shown in Table 4. For example, instance 1 has a pattern (3, 3, 8, 50, 3, 2, 1, 4), and the number of is according to the pool heuristics (eight features for the given example).


InstanceLower boundUpper boundNodesEdges

1338503214
2510636431.51.5
33825803124
4101050802431

6. Methodology for Determining a Subset of Heuristics

In this section, we propose a new approach for selecting and determining a subset of heuristics to solve GCP and CVRP instances. We describe our methodology in the next steps and the graphical representation of our methodology is shown in Figure 3.(1)Variables and Problem Restrictions Identification. First, the variables and restrictions of the problem are identified according to the problem aims or objectives. To model the GCP, the values in the MMA matrix represent the weights of the edges of nodes. If there is a zero in a certain position in the matrix, this represents no connection between those nodes. For graph coloring, each node is colored considering that the adjacent nodes do not have the same color. CVRP is aligned with our methodology due to its aim seeking to get subroutes in which the tour cost (subgroup) must be the minimum or the cheapest.(2)Problem’s Restriction Modeling. In both problems, we must design a partition of nodes or cities. First, it is necessary to model the restrictions for each variable in an LPH, e.g., a node cannot be colored by a specific color or a restricted city for a tour. Then, it must design the MMA which represents the edge or connection weight between nodes or cities. In GCP, the adjacency matrix corresponds to MMA, and in CVRP, the MMA matrix will be the matrix that has the distances of the node to node. For GC, our LPH is constructed based on the number of colors in which the nodes can be labeled. In case the problem restricts colors to five, the list will be like the one shown in Table 2. Similarly, this list will be built for the CVRP, where the number of vehicles is the number of parts that should be represented on the list (see Table 2). For the problems used in this work, it was not necessary to elaborate additional structures for soft restrictions. Besides, for an extensive review and how to model additional restrictions, the research proposed by Ortiz [10] details all possible cases and different features.(3)Apply the metalearning process described in Section 5.1(4)Separate the patterns (step 6) into training and test sets to proceed to the classification phase. It is important to consider at least one pattern of each class in the test set.(5)Use the classifier on the training set to make necessary adjustments to it. After describing and getting all pattern characteristics per instance, the next step is training and testing all instances by a classifier. For our approach, we prefer to use a simple classifier as Bayesian because our objective was not to compare the performance between classification algorithms or to design ad hoc classifiers for our research. The NBC simplifies learning by assuming per class that all features are independent [45]. In our methodology, we assume that each heuristic performance is independent because we applied each heuristic to independent experiments. In the previous stage, each experiment must be run with only one heuristic, and thus, we did not apply two or more heuristics at a time. Finally, all features in the created dataset instances were normalized before applying the classifier.(6)Finally, the set of test instances will use the classifier to assign a “class” and solve it with its corresponding set of heuristics.

6.1. Designing and Testing the Hyperheuristic Offline Learning with K-Folds

To choose the minimal set of heuristics and design the hyperheuristic for each class in more detail, our methodology considered the hyperheuristics with offline training as it has demonstrated good results for constraint satisfaction problems in terms of generality solution [3]. A random constructive heuristic was used to generate solutions to our problem of GCP, and for CVRP, a greedy algorithm was used. A selection hyperheuristic algorithm has three components: the pool of operators (low-level heuristics), a high-level search strategy, and a control mechanism to select the operator, which will be applied at each search step.

6.1.1. High-Level Search Strategy

The iterated local search algorithm was used as a high-level search strategy. This metaheuristic was proposed by Lourenço et al. [46] and it is constructing a sequence of solutions generated by an embedded heuristic. The generated solutions could be better if they were only constructed randomly. The essence of this algorithm is to intensify an initial solution, exploring neighboring solutions to it. The algorithm is shown in Algorithm 1, which was taken from El-Ghazali [47]. In the field of hyperheuristics with offline learning, it refers to the fact that the high-level search strategy searches for a methodology (a sequence of heuristics) that solves a set of instances and then applies it to a given set of instances, in contrast to online learning, which refers to the construction of a given sequence of heuristics as the instances are presented.

(1)s0=GenerateInitialSolution
(2)s∗=ImprovementStage(s0)
(3)while !StopCondition() do
(4)s'=SimpleRandomPerturbation(s∗)
(5)s'=ImprovementStage(s')
(6)if f(s∗') < f(s∗) then
(7)  s∗=s∗'
(8)end if
(9)end while
(10)return s
6.1.2. Selection Operator

In the perturbation phase (step 4 in Algorithm 2), it is necessary to choose a variable following a probability distribution based on the frequency of variable selection in the last iterations. This simple heuristic allows us to modify the methodology solution. We used the same hyperheuristic framework, and according to each class, we gave a different pool of low-level heuristics. For example, if for class 1 the best heuristics were , , , and , these were our pool for the hyperheuristic. With a similar procedure, the design was done for each class. After this process, we trained each hyperheuristic on their respective classes for the next stage.

(1)ls ← IncumbentSolution
(2)while! LocalStopCriteria() do
(3)hi=Perturbate()
(4)ls=apply(hi, ls)
(5)if f(ls) < f(ls) then
(6)  ls=ls'
(7)end if
(8)end while
(9)return ls

7. Experimental Results

This section describes our experiments in detail for graph coloring and CVRP benchmarks used in this paper. We give the configuration for the implementation of the iterated local search hyperheuristic. Finally, we described the statistical tests that we used to compare our results with the experimental methodology.

Our approach was implemented in JAVA language with JDK 1.8 using the IDE NetBeans IDE 8.2. The experiments were executed on a computer with processor Intel i7-7700U, 2.6 GHz, 16 GB DDR3 RAM, and operating system Windows 10 Home. The tests presented in this work were executed in a common notebook, with a single processor; it is showing the effectiveness of the exposed methodology.

For each heuristic, a limit of 100,000 function calls was given in each test run for all instances. We applied the Shapiro–Wilks test to check if the data results were normal or not, hence choosing a better representative (average or medium). If the data behavior is according to a normal distribution, the average was taken as representative and otherwise the median.

7.1. Heuristics Results for Graph Coloring and CVRP
7.1.1. Graph Coloring

We used the benchmark proposed for the second DIMACS challenge on graph coloring [48] and this is tested with 41 runs. In Tables 5 and 6, we show our results. We denote the best results with a bold face, and only the myciel2 instance was solved with the application of the individual heuristics in their optimum.


#NameNodesColors

11-Insertions_46731218111510666113252875
22-Insertions_33734745424154111029
32-Insertions_414933341163232148158294547155
43-Insertions_356359445332449940
54-Insertions_37921419161311616613252271
6anna138101717141716594513353068
7asH_3 31GPIA662399768596821067724891641561082
8asH_6 08GPIA121631497144872915411370200311201083151314852066
9asH_9 58GPIA1916326182553190026212462316119992039261626243206
10david87101618141616494815353758
11DSJC1000.1100019184718016101837178125651019942178918212634
12DSJC1000.51000822432251383824252379313011261230232723273234
13DSJC1000.910002141856207382818501824235412041062237517682414
14DSJC125.11254485936485416197375666173
15DSJC125.5125151131567211412427513368250112298
16DSJC125.9125421251737712413121013272208112221
17DSJC250.525025350415146349368658278157339326681
18DSJC250.925070311388152315319477234153487294485
19DSJC500.15001162964619763459810855322796236141114
20DSJC500.550047927100729091190513814713688598531412
21DSJC500.9500122771927326759763104448738210517311129
22DSJR500.1500111241303312511933412846129127343
23DSJR500.1c50084125714392031262125915661162475153912101589
24DSJR500.5500121376425201372369619299211627352662
25fpsol2.i.1496641411838213912226316991255262267
26fpsol2.i.245129528159515233728152204215348
27fpsol2.i.342529518255515436427254211232368
28games12012081624111717823410232388
29homer561122322222423165139245344176
30huck74101313121313463812272447
31inithx.i.1864537895818080423320115264263444
32inithx.i.2645305510062565952545985350324531
33inithx.i.3621305910865566351046278338331539
34jean8091312131412473113272646
35le450_15a45014292293103291291580410137292289597
36le450_15b45014289297106290285589427125300288624
37le450_15c4501471676235871371311878874247347071179
38le450_15d4501472375137171873311637994187287251197
39le450_25a450241601625015815337222963164162395
40le450_25b450241561615116416037718858156156404
41le450_25c45024407417150400393736510180396394780
42le450_25d45024400422150400403736461179397394791
43le450_5a450478077248475573811959665517537711209
44le450_5b450476976849875973811928865657647691280


#NameNodesColors

45le450_5c45041459147690914221394202416921083142814642073
46le450_5d45041464144284314491385198117121060143114362004
47miles1000128415258495149133754912970137
48miles150012873811078282811499479123100169
49miles25012871414151513733716242875
50miles500128192624252524874824814390
51miles750128303841383837102643710153119
52mugg100_1100351246555195121257
53mugg100_25100351246649254121455
54mugg88_18835104554319591050
55mugg88_2588351045646205111161
56mulsol.i.1197486068606057142765913180153
57mulsol.i.21883049564850471721184814395186
58mulsol.i.31843050564849471771174816099176
59mulsol.i.41853049535049481651194814294192
60mulsol.i.518630506248505017111448107100191
61myciel311346433106410412
62myciel4234596552412515927
63myciel547581389853328201756
64myciel69561824121919135105133751143
65myciel71917849618857834730323107136328
66qg.order3090029554544152554525912223227533531953
67qg.order4016003911741105627116010781626378558113111281671
68qg.order603600593076290724253079289736928312234307030083738
69queen10_1010010415837383515565376071163
70queen11_11121105776545750209101538782216
71queen12_1214412517452544922493488174233
72queen13_13169127010767746828111269108117299
73queen14_141961558835456532881175393101317
74queen15_15225157511171817437013874124112368
75queen16_16256168510883908040715979125117438
76queen5_52541738161715452517252350
77queen6_63661927171718492816472755
78queen7_74962951273033904230363987
79queen8_129611295230322513860284848138
80queen8_8648325625313310045278042104
81queen9_9819314833302712455344958129
82school1385139971008626100998714231330691107610371440
83school1_nsh35213723742445735720110010484907347811135
84will199GPIA7013438418438453430176215424425475551788
85zeroin.i.1211486272626160147806012689159
86zeroin.i.22112942554141411651134516389169
87zeroin.i.32062940554340391791114016291164

We applied a nonparametric test to verify that there are differences between the performance of each heuristic. Table 7 shows the ranges obtained in the three statistical tests for graph coloring instances. The three omnibus tests indicated there are significant differences between the heuristics. The heuristics which obtained the highest ranks in the three tests were and , i.e., these have the worst performance. and were the best results for the problems, and this is because in all omnibus tests those heuristics obtained the lowest ranks (see Table 7 marked in bold). The time of each run is reported in Table 8 in.


Statistics value

F255.7204.646.402.184.683.3310.017.522.487.406.4710.89
FA77.860374.88461.35209.30370.94340.22820.66575.50212.84568.01505.86829.45
Q99.0405.276.491.915.243.689.996.842.447.116.1310.90


Instance

1-Insertions_4151418151411136151413
2-Insertions_3141417141411127141413
2-Insertions_4161617171612147161614
3-Insertions_3141417141411126141413
4-Insertions_3151418151411157151513
anna16151516151295161614
asH_3 31GPIA5844314644436127464745
asH_6 08GPIA2011728117817515815693160161134
asH_9 58GPIA488321140370354371317203371371275
david15141415141195151514
DSJC1000.16762256463732430636831
DSJC1000.55153215559481723595921
DSJC1000.94955215361481624736522
DSJC125.1151517161512126151514
DSJC125.515151516151294151514
DSJC125.915151316151284161613
DSJC250.518181419181496191915
DSJC250.918181419181485201914
DSJC500.13027173028241511292920
DSJC500.5262615282722109292915
DSJC500.925261527282299333115
DSJR500.13027172928251511292920
DSJR500.1c26261528282299313015
DSJR500.525261527282299333115
fpsol2.i.12726152927241310303315
fpsol2.i.2252515262521109262716
fpsol2.i.3242515252420108252615
games120151515161512105151514
homer3230183334271712323424
huck14141515141195141414
inithx.i.15249225652511924547124
inithx.i.23334173533301513363718
inithx.i.33234173432291412353617
jean14151515141195151414
le450_15a252616272522129272817
le450_15b252716272522129272817
le450_15c252716272522129262817
le450_15d252716272522129272817
le450_25a242515262521109272716
le450_25b242515262521109272715
le450_25c252615262521109272716
le450_25d252515272521109272716
le450_5a2631202826232512273222
le450_5b2631202826232412273222
le450_5c2529202725222412272822
le450_5d2529202725222412262822
miles1000151513161512841161613
miles1500151615161512841151613
miles250151615161512105151613
miles500151514161512841161514
miles75015151316151284161613
mugg100_1151517151511137151513
mugg100_25151517151511137151513
mugg88_1151517151511136151513
mugg88_25151517151511137151513
mulsol.i.117181418171385181814
mulsol.i.216181417161385171714
mulsol.i.316181417161385171714
mulsol.i.416181417161385171714
mulsol.i.516181417161385171714
myciel3141417141410126141413
myciel4141416141410116141413
myciel5141416141410115141413
myciel6141515151511115151514
myciel7161716171713116171714
qg.order305053215451492023545726
qg.order4014915146157152187447115916753
qg.order6010161016184983108798319153210991113226
queen10_1015151515151195151514
queen11_1115151515151295161614
queen12_1216151416151295161614
queen13_1316161517161295161614
queen14_1417171518171395171715
queen15_1518181519181496181815
queen16_1618191419191596191915
queen5_5141418141410126141414
queen6_6141416141410105141414
queen7_7141416141411115141414
queen8_1215151415141195151514
queen8_8141416151411105141414
queen9_9141414151411105151514
school1232416242319128242517
school1_nsh212216222118117222316
will199GPIA4751355967516832545246
zeroin.i.117181418171485181814
zeroin.i.217181518171495181814
zeroin.i.317181518171395181814

7.1.2. Capacitated Vehicle Routing Problem (CVRP)

Three sets of the state of the art were used and tested on 41 runs:(1)Augerat et al. (SET A), 9 instances, proposed in [49](2)Christofides, Mingozzi, and Toth (CMT), 14 instances, proposed in [50](3)Golden, Wasil, Kelly, and Chao (GWKC), 20 instances, proposed in [51](4)Uchoa et al., 9 instances, proposed in [52]

In Table 9, we show the fitness values for the instances and the lowest city cost tour is indicated in bold, where is the number of nodes, is the capacity of each vehicle, and is the number of vehicles (colors in the case of graph coloring). The time of each run is reported in Table 10.


InstanceCitiesVehiclesOP

88CMT1505524.6115831161156915681117114116641576166416641664
89CMT27510835.2623531591235123491728189825772547257725772577
90CMT31008826.1419041821193920171699174332932249329332933293
91CMT4150121028.4225782674252025192347246250963115509650965096
92CMT5199171291.2931594469649031443008315567426707674266926742
93CMT6506555.43102711601569102894193316641573166415711664
94CMT77511909.6814921616235114651357140725772517257725312577
95CMT81009865.9517431890179919061598163632932203329331953293
96CMT9150141162.5523082768248923252185225650962971509649185096
97CMT10199181395.8546074607285228522791287067426707674265136742
98CMT1112071042.1220772077237223722046211026082449260826082608
99CMT1210010819.5616921692173317331472152823062144228723002306
100CMT13120111541.1420402040204120411827190626082476260825772608
101CMT1410011866.3717171717160916091444151723062171230622702306
102GWKC124095623.479497915586079261735085651331210546133121331213312
103GWKC2320108404.611457116631151611339912320109032096116801207872073521080
104GWKC3400911036.21919721734241612007017167187932611722304261172611726117
105GWKC448010135902704630204323822664327133265043873231793388923889238892
106GWKC520056460.981422415677130091420712800134351955815763213782137821378
107GWKC628078412.91540721043149251570114022145392669917688266992669926699
108GWKC7360810102.72590923841259092590924611257182590925909259092590925909
109GWKC84401011635.32150023969208172048318560186773150626204315063150631506
110GWKC925514579.7111029892323126495491625812319258125812581
111GWKC1032316735.6613441312347612771222122935893451358935893589
112GWKC1139917912.0317511919475816401572148550404478503950395040
113GWKC12483191101.522951979673824622192236354166520675067506750
114GWKC1325226857.1920101301206220311475161521172078215821512158
115GWKC14320291080.5523511637263723222221238927482665274527442748
116GWKC15396331337.8732622062332232072793302533663308336633643366
117GWKC16480361611.5639782596410738203492378341984092421742124217
118GWKC1724022707.7617541518216317291611166923762165237623762376
119GWKC1830027995.1321302165344321502073215534433443344334253443
120GWKC19360331365.632382808454030243226357547134466471347134713
121GWKC20420381817.5935193746619035113189332761865992619160846191
122A-n32-k532578415381476143114741265125522301662222922292230
123A-n45-k64561733.3617331733173317331273115617331733173317331733
124A-n55-k9559107316601413145316421439132823761901237623762376
125A-n61-k9619103425001814233824641731183825922517259225922592
126A-n62-k8628128817721724203917921551161531942219319431943194
127A-n63-k9639131424801879245223961813183526772564267726772677
128A-n64-k9659140119351893216919481744171526132203261326132613
129A-n65-k9659117426091940260025461820189229492808294929492949
130A-n69-k9699115921491803208019791626168732202477322032203220
131X-n148-k4614846434487182654940750627166568924711797762378715791167911679116
132X-n153-k2215322212203670039843527413615534116343377180156820717997179371801
133X-n157-k1315713168763636424507363643636427917292103636436364363643636436364
134X-n162-k1116211141383923237225363813816232918340597997360657799697996379973
135X-n367-k173671722814712098532697502724826864870027154244112919154244154244154244
136X-n393-k38393393826014143274629152513143398130939148728158058155203158058158058158058
137X-n401-k294012966187140100106590163158141582141056145043197323163230197323197323197323
138X-n411-k19411191971867963796428103467090630386617317326988152173269173269173269
139X-n420-k13042013010798182376132745207213183053203757200726217483216871217483217483217483



A-n32-k5771277574779
A-n45-k6771277563779
A-n55-k9771177553779
A-n61-k9771177553779
A-n62-k8771187563779
A-n63-k9771177553779
A-n63-k10771177553779
A-n64-k9771177553779
A-n65-k9771177553779
A-n69-k9771187553879
CMT1771277574779
CMT2771177553889
CMT3771187563889
CMT4881188663999
CMT588109865310109
CMT6771277563779
CMT7771187552879
CMT8771187563889
CMT9881088652999
CMT1088109865210109
CMT11771187663889
CMT12771277573779
CMT13771277563779
CMT14771277574779
X-n420-k13011111611119118111113
X-n411-k1911111611119118111113
X-n401-k2911111611119107111113
GWKC1881198663999
GWKC28811108663101010
GWKC39911119784121213
GWKC41010131210883121211
GWKC5881398694989
GWKC68812986739910
GWKC7991199773101010
GWKC89911119773121210
GWKC9881198653101010
GWKC109911109763121110
GWKC119911119763131310
GWKC121010111210863151511
GWKC138811109752121210
GWKC149911109753141410
GWKC15101011119753161610
GWKC161010111210853191810
GWKC17881098652111110
GWKC189910109752131310
GWKC199911119753161510
GWKC201010111210853181811
X-n148-k4611111611119118111113
X-n153-k2211111611119117111113
X-n157-k1311111611119107111113
X-n162-k1111111511119107111113
X-n367-k1711111511119107111113
X-n393-k3811111511119107111113

We applied the same procedure to the statistical tests of Friedman (FT), Alienated Friedman (AFT), and Quade (Qt) to distinguish the behavior of the heuristics set. We established and as there are no differences between the performance of the heuristics and established as there are differences between the performance of the heuristics. Table 11 shows the ranks obtained in the three statistical tests.


Statistics value

F420.8404.498.799.693.535.594.161.602.469.326.929.45
FA47.30170.82441.25443.96143.53280.22169.32127.19147.94441.70342.00443.57
Q79.9704.298.879.723.685.593.961.672.579.266.919.47

In this case, the heuristic has the lowest rank for the tests and has the second-lowest rank for QT and FT.

7.2. Selection of Features and Classes by Statistical Tests

According to the steps mentioned in Section 5.1, we must determine first the number of clusters or classes to split all our test instances. In this case, and .

We considered 8 classes and used -means clusters and we expected uniformly distributed instances in the clusters. The -means algorithm was applied with a maximum number of , initially random starting points. To consider the uniform distribution of classes into clusters, we used the Manhattan distance obtained after the experimental work, with the best results.

Table 12 contains the class details, number of instances per cluster/class, number of GCP (3rd column), or CVRP (4th column) per class, min and max nodes, and min and max number of colors nodes. In this experimentation, clusters 1, 5, 6, and 7 have only GCP instances, clusters 3, 4, and 8 have CVRP instances, and only cluster 2 has both problem domains.


Cluster# of instances%GCCVRPMin nodesMax nodesMin colors/vehiclesMax colors/vehicles

11812.91801114936
2107.28242036003214
3117.9011148480546
453.60536742017130
51913.719064256730
63122.331012590011122
7117.9110352701314
83424.503432483533

7.3. Training and Test Classifiers for the Instance’s Classes

After the heuristic pool design phase for the hyperheuristics, we split our dataset into training and test. The training dataset was created by 125 instances with 15 features (basic + inner) and the unseen instances were made by 15 instances. The results of the classification with Naive Bayes are reported in Table 13.


Correctly classified11491.20%
Incorrectly classified118.80%
TP rate0.912
FP rate0.022
Precision0.918
Recall0.912

Table 14 contains the confusion matrix of the process classification. We observed that, for some classes like 3, 4, 7, and 8, the patterns were classified correctly. The rest of the classes have some patterns classified incorrectly, but, e.g., for the 3 patterns of class 1 classified into 5 and 6, we used the same pool of low-level heuristics and this does not represent an issue for the next step.


Class12345678TPRFPRP

11300021000.81301
2070000020.77801
3001000000101
400040000101
50000125000.7060.0190.857
60000027100.9640.0620.818
700000010010.0090.909
800000003110.0210.939

7.4. Designing and Testing the Hyperheuristic Offline Learning with K-Folds

In the next step, the statistical tests were applied to heuristics and will form the characteristics of our instances, graph coloring, and CVRP per class. In this phase, we choose according to the rankings the heuristics which have . If, for example, the Aligned Friedman has a min rank and max rank , the limit value for considering a heuristic must be . Because heuristic 11 has the worst performance for both datasets, we did not consider this heuristic for this experimentation phase.

Table 15 shows the ranks and the heuristic fitness for each class, e.g., class 1 for QT, and AFT has the same heuristics , , , , , and , while FT does not consider . We only consider , , , , , and as a minimum set.


Class 1Class 2

Fr
1.501.693.784.034.086.692.833.445.76.25

A-Fr
6276381089109111021760.5293320335428508540

Qu
1.551.593.774.044.156.812.242.732.984.185.516.39
Class 3Class 4

Fr
1.642.184.274.274.825.182.42.62.83.63.66

A-Fr
2422834034184195683374787991128

Qu
1.712.264.204.524.885.382.072.672.673.673.936
Class 5Class 6

Fr
1.451.553.534.374.686.031.321.694.354.925.055.92

A-Fr
6466511144.511971225.52010.513061345411242114227.505058

Qu
1.471.533.674.444.695.781.481.533.904.554.606.47
Class 7Class 8

Fr
1.322.453.455.325.865.911.472.383.444.214.595.46

A-Fr
108.50171477.50578.50637.50642232826912989384740706169.50

Qu
1.122.173.655.095.745.821.502.283.274.464.705.43

The selected heuristics for each class can be summarized into five groups:(1), , , , and (2), , , , and (3), , , , , and (4), , , , and (5), , , and

Therefore, we design 5 algorithms for these 8 classes. This means the hyperheuristic with the high-level search strategy was the same, but the heuristics pool set was different according to these 5 subset heuristics. We trained and tested the hyperheuristic on short-length pools. We left 10% of instances as unseen for the hyperheuristics and the results are shown in Table 16.


#CMedianDEAverageOpt#CMedianDEAverageOpt#CMedianDEAverageOpt

413243305104121077222841623
1143930334510512919716021249011
314129349581015737732618562314
4135535052011261938732118362014
5131073515321440304374071606744
14118284046557737274474021686724
5313863695222536104572943981,1474
5413553705373356104672933901,1564
55136537153242551283734017165813
6113033725525579128473204135483
62142547353558641512381,8212611,5241,733
631567575560801011612481,1475171,5981,073
641624196805151525812581,3664682,0821,034
7614784815182129912681,4107672,0521,288
771881261666033994212781,5824372,1451,314
7811611236186127822557012881,5104551,9751,401
825733651,17632062472857154712981,4335832,2711,174
921,0115102,190321628520662812213081,4207472,1201,159
1125415351,43519226215996118889092911,339525
1327714781,542214246181973181219782,4691,8974,6221,396
232153441962842567543133649881,7073422,1741,042
11722,1797763,3431,612266427798299981,1164261,605820
12122,8581,5204,5281,81827640801002910081,5484182,0511,541
672318330897392961435351210181,1174201,564866
6827536892,522593166886131539081,4417872,105826
10339,4294,58713,9678,40532645130142309182,0491,3133,0991,028
104313,2115,61719,24011,03633646136145309282,6191,8424,7021,291
106311,3764,02514,7626,46135685104239149388153421,165555
107312,5666,11817,6968,41336685108238149481,1885721,800910
108313,2214,90523,10710,1033963768126249581,4367272,024866
109314,3907,05521,41111,63540635691242410286,0233,0218,8745,624
131350,6349,12668,65643,4484261271373322411181,0561,1722,201736
132330,32317,17347,02521,2204764315534111281,3401,6712,996912
133322,1755,42032,88316,8764867415867311381,7442,2874,0381,102
134328,76921,92245,67814,1385665017624811481,1384221,658857
135465,09859,475160,61722,8145764028593011581,4475222,1541,081
136467,18429,544127,82738,2605863726573011681,8115742,7371,338
137497,07434,915149,75066,1876064027583011881,3004621,810708
1394125,87527,756192,09310,7986661311934172911981,6238372,587995
10510615108565317654812082,3701,0323,5191,366
155454895158663428552911088238031,603580
28581414887633265329

The best results are highlighted in bold.

The hyperheuristic configuration was 10 iterations for local search and 100,000 function calls. For some GCP instances, we got the optimum number of colors (denoted in bold in Table 16). Besides, for the instances A-n45-k6, A-n55-k9, A-n62-k8, A-n64-k9, CMT13, GWKC1, GWKC2, GWKC3, and X-n148-k46, we get values near to the optimal with a maximum of 20% of the distance.

7.5. Classification of the Test Instances and Application of the Hyperheuristic to the Corresponding Instance

Finally, for the 14 unseen instances, we used the Naive Bayes classifier, and it determines the class for these instances. Later, we applied the hyperheuristic with the corresponding pool heuristics according to the previous design and we obtained the results shown in Tables 17 and 18.


Class12345678TPRFPRP

120000000101
201000000101
300100000101
400010000101
50000020000
60000030010.1820.6
700000010101
800000003101


InstanceClassFitnessInstanceClassFitness

anna13DSJC250.5627
mugg100_113le450_25c626
DSJC1000.5283mulsol.i.4630
GWKC4314255.68school1713
X-n411-k19420005.37A-n32-k58876.19
queen15_15529CMT28911.37
queen8_12521CMT981258.57

Table 17 shows the confusion matrix, TP rate, FP rate, and precision of the classification test. In these results, two patterns that belong to class five were classified incorrectly, but this does not affect the hyperheuristic solution because this class shares the same heuristics with class 6.

7.6. Statistical Comparison of Results

Finally, to compare if there are differences between the results applying the methodology and without applying the methodology, an experiment was carried out where the hyperheuristics were executed 33 times with the entire set of heuristics and 100,000 function calls. The results are shown in Table 19.


InstancecHHPCInstancecHHPCInstancecHHPCInstancecHHPC

1-Insertions_4134X-n367-k174155287le450_15b6242CMT1282345
2-Insertions_3111X-n393-k384158696le450_25a6137CMT1382769
2-Insertions_4160X-n401-k294198461le450_25b6140CMT1482361
3-Insertions_3114X-n420-k1304218854le450_25d6341CMT383335
4-Insertions_3128david527miles1000672CMT485185
DSJC125.1170DSJC125.55138miles1500697CMT586785
mugg100_25115games120531mulsol.i.1674CMT681671
mugg88_1120huck522mulsol.i.2687CMT782582
mugg88_25114jean520mulsol.i.3674CMT883355
myciel316miles250524mulsol.i.5670GWKC1813602
myciel419miles500541qg.order306351GWKC1083630
myciel5126miles750551zeroin.i.1678GWKC1185057
myciel6148myciel75117zeroin.i.2670GWKC1286775
queen5_5126queen10_10582zeroin.i.3678GWKC1382191
queen6_6129queen11_115108asH_3 31GPIA7433GWKC1482793
queen7_7142queen12_125105DSJC500.17477GWKC1583413
asH_6 08GPIA2746queen13_135141le450_15c7618GWKC1782398
asH_9 58GPIA21157queen14_145143le450_15d7643GWKC1883467
DSJC1000.121090queen16_165184le450_5a7632GWKC1984756
DSJC1000.921625queen8_8550le450_5b7699GWKC982604
DSJR500.1c2930queen9_9569le450_5c7861A-n32-k5T2235
GWKC1624264DSJC125.96119le450_5d7792annaT27
GWKC2026248DSJC250.96255school1_nsh7671CMT2T2611
qg.order402623DSJC500.56656will199GPIA7898CMT9T5145
qg.order6022098DSJC500.96611A-n45-k681856DSJC1000.5T1765
GWKC2321504DSJR500.16100A-n55-k982474DSJC250.5T336
GWKC3326716DSJR500.56314A-n61-k982681GWKC4T39592
GWKC5321807fpsol2.i.16137A-n62-k883200le450_25cT321
GWKC6327213fpsol2.i.26136A-n63-k982774mugg100_1T17
GWKC7326506fpsol2.i.36131A-n64-k982709mulsol.i.4T104
GWKC8332073homer650A-n65-k982977queen15_15T164
X-n148-k46380822inithx.i.16190A-n69-k983258queen8_12T52
X-n153-k22372255inithx.i.26188CMT181709school1T850
X-n157-k13337747inithx.i.36178CMT1086763X-n411-k19T174414
X-n162-k11380314le450_15a6251CMT1182624

First, the statistical distributions followed by each set of results by class were analyzed, that is, the Shapiro–Wilks test [53] was applied to determine if the results of the methodology, hyperheuristic without methodology (HHPC), and the optimal state of the art followed a normal distribution. The test was applied with a ∝ = 0.05. The results of the test are shown in Table 20, and the data shown in Tables 16, 18, and 19 were taken for the tests. It should be noted that the results of the methodology only for clusters 4 and 5 were normal, the results of the optimal state of the art only for only cluster 4 were normal, and the results of HHPC for clusters 1, 4, 5, and 7 were normal.


ClusterMethodologyState of the artHHPC

12.26E − 052.69E − 041.13E − 01
20.03091.57E − 047.52E − 03
30.014381.80E − 034.28E − 03
40.35517.91E − 013.29E − 01
50.054661.05E − 039.61E − 02
61.20E − 056.53E − 052.25E − 05
70.02553.32E − 037.31E − 01
84.95E − 082.40E − 094.59E − 07

Student’ t-tests for methodology and state of the art were applied. For the state of the art, methodology, and HHPC, we established ∝ = 0.05 as a level of significance. The null and alternative hypotheses for methodology and HHPC are as follows:(i): there are no differences between the performance of hyperheuristic with the methodology and without the methodology(ii): there are differences between the performance of hyperheuristics with the methodology and without the methodology

For methodology and state of the art,(i): there are no differences between the performance of hyperheuristics and the optimal state of the art(ii): there are differences between the performance of hyperheuristics

The statistical results of the tests are shown in Table 21. With these values, we can observe the following:(i)Methodology and HHPC. It can be inferred that the results of the methodology are significantly different from those of hyperheuristics with the whole set of heuristics. This means that the methodology improved performance and allowed limiting the set of heuristics for each of the clusters.(ii)Methodology and State of the Art. It can be inferred that no statistical evidence was found that the results of the methodology differ from the optimal ones of the state of the art, except in clusters 5 and 7. This is because it is where there are more atypical data or that they were badly classified which opens an area of opportunity for the refinement of the methodology.


ClusterMethodology and Edo. Art
Methodology and HHPC

10.16415.12E-05
20.14890.1496
30.32110.02586
40.028140.004389
50.006010.0002388
60.011440.0009709
74.43E-076.65E-06
80.062352.52E-05

8. Conclusion

In this work, a methodology is proposed to select low-level heuristics for a hyperheuristic approach to offline learning oriented to the solution of instances of different constraint satisfaction problems. The proposal was applied to two different problems well known and studied in the state of the art, which were the coloring of graphs and the vehicle routing problem with a specific capacity, GCP and CVRP, respectively.

The methodology is focused on optimizing the number of heuristics that can be applied to different constraint satisfaction problems in a hyperheuristic approach. Information on the performance of an original set of heuristics for the instances of the problem is obtained from the different problems. The performance information is used to generate characteristic vectors for each instance, which is used to generate equivalence classes of instances of the problem. The grouping in classes allows to identify the heuristics that apply to each class and from that information, a reduction of the number of heuristics necessary to obtain good solutions in the instance of each class is made and to reduce the total number of heuristics that can be applied in the hyperheuristic approach to solving the problems involved.

In the application to the GCP and CVRP, the information on the performance of the heuristics was obtained through a metalearning process, and this information was used to obtain the basic and internal characteristics of the instances. The instances were grouped into 5 classes using the k-means algorithm with the Mahalanobis metric. For each class, the sets of heuristics that could be applied to all their instances were identified, and through a process of hierarchization and cutoff criteria, the number of heuristics per class was reduced.

For training and testing, the Naive Bayes classifier and information on the characteristics of the instances were used. The experimental results show that the hyperheuristic in each class could efficiently solve each instance, and the classifier was able to predict the class for each problem instance.

The identification and reduction of heuristics to find the solution of complex problems is an optimization strategy that can do the search for solutions to problems of satisfaction of restrictions efficiently. The methodology presented allows generating a framework with a level of generality that can be trained to solve different problems of satisfaction of constraints simultaneously under the hyperheuristic approach. Once trained, it can allow finding good solutions to different problems with a common base of heuristics for instances of problems grouped by the efficiency of the solution heuristics.

Finally, the methodology makes it possible to improve the search for solutions to sets of problems by exploring the diversification of some of its components such as classification algorithms, metrics, heuristics, and selection criteria, which may be different for sets of different problems. A study of these possibilities is proposed as future work.

Data Availability

The instances data used to support the findings of this study have been deposited in the graph coloring repository http://vrp.atd-lab.inf.puc-rio.br/index.php/en/, http://archive.dimacs.rutgers.edu/pub/challenge/graph/benchmarks/color/, https://neo.lcc.uma.es/vrp/vrp-instances/capacitated-vrp-instances/. They are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

The authors thank Tecnológico Nacional de México/I. T. León and Universidad de Guanajuato. This work was supported by the National Council of Science and Technology of Mexico (CONACYT) via the Scholarship for Postgraduate Study 446106 (L. Ortiz) and Research Grant: CÁTEDRAS-2598 (A. Rojas). The authors thank the participation of Valentin Calzada-Ledesma from Instituto Tecnológico Superior de Purísima for the revision and correction of this article.

References

  1. D. H. Wolpert and W. G. Macready, “No free lunch theorems for optimization,” IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, 1997. View at: Publisher Site | Google Scholar
  2. N. Pillay, “A review of hyper-heuristics for educational timetabling,” Annals of Operations Research, vol. 239, no. 1, pp. 3–38, 2016. View at: Publisher Site | Google Scholar
  3. W. B. Yates and E. C. Keedwell, “An analysis of heuristic subsequences for offline hyper-heuristic learning,” Journal of Heuristics, vol. 25, no. 3, pp. 399–430, 2019. View at: Publisher Site | Google Scholar
  4. E. K. Burke, M. R. Hyde, G. Kendall et al., “A classification of hyper-heuristic approaches: revisited,” in Handbook of Metaheuristics, pp. 453–477, Springer International Publishing, Berlin, Germany, 2018. View at: Publisher Site | Google Scholar
  5. J. Kanda, A. D. Carvalho, E. Hruschka, C. Soares, and P. Brazdil, “Meta-learning to select the best meta-heuristic for the traveling salesman problem: a comparison of meta-features,” Neurocomputing, vol. 205, pp. 393–406, 2016. View at: Publisher Site | Google Scholar
  6. J. R. Rice, “The algorithm selection problem,” in Advances in Computers, pp. 65–118, Elsevier, Amsterdam, Netherlands, 1976. View at: Publisher Site | Google Scholar
  7. I. Amaya, J. C. Ortiz-Bayliss, S. Conant-Pablos, and H. Terashima-Marin, “Hyper-heuristics reversed: learning to combine solvers by evolving instances,” in Proceedings of the 2019 IEEE Congress on Evolutionary Computation (CEC), IEEE, Wellington, New Zealand, December 2019. View at: Publisher Site | Google Scholar
  8. H. Song, I. Triguero, and E. Özcan, “A review on the self and dual interactions between machine learning and optimisation,” Progress in Artificial Intelligence, vol. 8, no. 2, pp. 143–165, 2019. View at: Publisher Site | Google Scholar
  9. J. A. Montesino-Guerra, H. Puga, J. M. Carpio, M. Ornelas-Rodríguez, A. Rojas-Domínguez, and L. Ortiz-Aguilar, “Combinatorial designs on constraint satisfaction problem (VRP),” in Intuitionistic and Type-2 Fuzzy Logic Enhancements in Neural and Optimization Algorithms: Theory and Applications, pp. 509–526, Springer, Cham, Switzerland, 2020. View at: Google Scholar
  10. L. D. M. Ortiz-Aguilar, M. Carpio, H. Puga et al., “Increase methodology of design of course timetabling problem for students, classrooms, and teachers,” in Nature-Inspired Design of Hybrid Intelligent Systems, pp. 713–728, Springer, Berlin, Germany, 2017. View at: Google Scholar
  11. S. Lin and B. W. Kernighan, “An effective heuristic algorithm for the traveling-salesman problem,” Operations Research, vol. 21, no. 2, pp. 498–516, 1973. View at: Publisher Site | Google Scholar
  12. K. Helsgaun, “General k-opt submoves for the Lin–Kernighan TSP heuristic,” Mathematical Programming Computation, vol. 1, no. 2-3, pp. 119–163, 2009. View at: Publisher Site | Google Scholar
  13. A. Blazinskas and A. Misevicius, “Combining 2-opt, 3-opt and 4-opt with k-swap-kick perturbations for the traveling salesman problem,” in Proceedings of the 17th International Conference on Information and Software Technologies, pp. 27–29, Kaunas, Lithuania, April 2011. View at: Google Scholar
  14. S. Minton, M. D. Johnston, A. B. Philips et al., “Minimizing conflicts: a heuristic repair method for constraint satisfaction and scheduling problems,” Artificial Intelligence, vol. 58, no. 1–3, pp. 161–205, 1992. View at: Publisher Site | Google Scholar
  15. B. S. Baker, “A new proof for the first-fit decreasing bin-packing algorithm,” Journal of Algorithms, vol. 6, no. 1, pp. 49–70, 1985. View at: Publisher Site | Google Scholar
  16. J. Csirik, “The parametric behavior of the first-fit decreasing bin packing algorithm,” Journal of Algorithms, vol. 15, no. 1, pp. 1–28, 1993. View at: Publisher Site | Google Scholar
  17. J. A. Soria-Alcaraz, G. Ochoa, M. Carpio et al., “Effective learning hyper-heuristics for the course timetabling problem,” European Journal of Operational Research, vol. 238, no. 1, pp. 77–86, 2014. View at: Publisher Site | Google Scholar
  18. J. A. Soria-Alcaraz, E. Özcan, J. Swan, G. Kendall, and M. Carpio, “Iterated local search using an add and delete hyper-heuristic for university course timetabling,” Applied Soft Computing, vol. 40, pp. 581–593, 2016. View at: Publisher Site | Google Scholar
  19. W. B. Yates and E. C. Keedwell, “Offline learning for selection hyper-heuristics with Elman networks,” in Lecture Notes in Computer Science, pp. 217–230, Springer International Publishing, Berlin, Germany, 2018. View at: Google Scholar
  20. J. A. Soria-Alcaraz, G. Ochoa, M. A. Sotelo-Figeroa, and E. K. Burke, “A methodology for determining an effective subset of heuristics in selection hyper-heuristics,” European Journal of Operational Research, vol. 260, no. 3, pp. 972–983, 2017. View at: Publisher Site | Google Scholar
  21. C. Martin, P. Héctor, T. M. Hugo et al., “Methodology of design: a novel generic approach applied to the course timetabling problem,” in Soft Computing Applications in Optimization, Control, and Recognition, pp. 287–319, Springer, Berlin, Germany, 2013. View at: Google Scholar
  22. P. Brazdil, C. Giraud-Carrier, C. Soares et al., Metalearning, Springer, Berlin, Germany, 2009.
  23. A. E. Gutierrez-Rodríguez, S. E. Conant-Pablos, J. C. Ortiz-Bayliss, and H. Terashima-Marín, “Selecting meta-heuristics for solving vehicle routing problems with time windows via meta-learning,” Expert Systems with Applications, vol. 118, pp. 470–481, 2019. View at: Publisher Site | Google Scholar
  24. R. Karp, Complexity of Computer Computations, Springer US, Boston, MA, USA, 1972.
  25. A. Kosowski and K. Manuszewski, “Classical coloring of graphs,” Graph Colorings, vol. 352, pp. 1–19, 2004. View at: Publisher Site | Google Scholar
  26. G. B. Dantzig and J. H. Ramser, “The truck dispatching problem,” Management Science, vol. 6, no. 1, pp. 80–91, 1959. View at: Publisher Site | Google Scholar
  27. E. Alba and B. Dorronsoro, “Computing nine new best-so-far solutions for capacitated VRP with a cellular genetic algorithm,” Information Processing Letters, vol. 98, no. 6, pp. 225–230, 2006. View at: Publisher Site | Google Scholar
  28. J. A. Soria-Alcaraz, M. Carpio, P. Hector et al., “Comparison of metaheuristic algorithms with a methodology of design for the evaluation of hard constraints over the course timetabling problem,” Studies in Computational Intelligence, vol. 451, Springer, Berlin, Germany, 2013. View at: Google Scholar
  29. M. Carpio, “Modelo integral de asignación optima de carga academica usando un algoritmo heurístico,” in Actas de la Encuentro de Investigación en Ingenieria Electrica, Zacatecas, Mexico, Febrero 2006. View at: Google Scholar
  30. L. D. M. Ortiz Aguilar, “Diseño de horarios de alumnos y maestros mediante técnicas de soft computing, para una institución educativa,” Instituto Tecnológico de León, León, Mexico, 2016, Master’s thesis. View at: Google Scholar
  31. E. Alpaydin, Introduction to Machine Learning, The MIT Press, Cambridge, MA, USA, 3rd edition, 2014.
  32. P. Brazdil and C. Giraud-Carrier, “Metalearning and algorithm selection: progress, state of the art and introduction to the 2018 special issue,” Machine Learning, vol. 107, no. 1, pp. 1–14, 2018. View at: Publisher Site | Google Scholar
  33. J. Sakarovitch, “Kleene’s theorem revisited,” in International Meeting of Young Computer Scientists, pp. 39–50, Springer, Berlin, Germany, 1986. View at: Google Scholar
  34. R. J. Campello, D. Moulavi, and J. Sander, “Density-based clustering based on hierarchical density estimates,” in Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 160–172, Springer, Berlin, Germany, 2013. View at: Google Scholar
  35. D. W. Scott, “Sturges’ rule,” Wiley Interdisciplinary Reviews: Computational Statistics, vol. 1, no. 3, pp. 303–306, 2009. View at: Publisher Site | Google Scholar
  36. J. Jokinen, T. Raty, and T. Lintonen, “Clustering structure analysis in time-series data with density-based clusterability measure,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 6, pp. 1332–1343, 2019. View at: Publisher Site | Google Scholar
  37. X. Xu, J. Li, M. Zhou, J. Xu, and J. Cao, “Accelerated two-stage particle swarm optimization for clustering not-well-separated data,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 50, no. 11, pp. 4212–4223, 2020. View at: Publisher Site | Google Scholar
  38. J. Derrac, S. García, D. Molina, and F. Herrera, “A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms,” Swarm and Evolutionary Computation, vol. 1, no. 1, pp. 3–18, 2011. View at: Publisher Site | Google Scholar
  39. D. de Werra, “Heuristics for graph coloring,” in Computational Graph Theory, pp. 191–208, Springer, Berlin, Germany, 1990. View at: Google Scholar
  40. H. Derbel, B. Jarboui, and R. Bhiri, “A skewed general variable neighbourhood search algorithm with fixed threshold for the heterogeneous fleet vehicle routing problem,” Annals of Operations Research, vol. 272, no. 1-2, pp. 243–272, 2019. View at: Publisher Site | Google Scholar
  41. M. M. Solomon, “On the worst-case performance of some heuristics for the vehicle routing and scheduling problem with time window constraints,” Networks, vol. 16, no. 2, pp. 161–174, 1986. View at: Publisher Site | Google Scholar
  42. S. S. Habashi, C. Salama, A. H. Yousef et al., “Adaptive diversifying hyper-heuristic based approach for timetabling problems,” in Proceedings of the 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pp. 259–266, IEEE, Vancouver, Canada, November 2018. View at: Google Scholar
  43. S. Abdullah, E. K. Burke, and B. McCollum, “Using a randomised iterative improvement algorithm with composite neighbourhood structures for the university course timetabling problem,” in Metaheuristics, Operations Research Computer Science Interfaces Series, vol. 39, pp. 153–169, Springer US, Boston, MA, USA, 2007. View at: Google Scholar
  44. I. P. Gent, M. Ewan, P. Patrick et al., “An empirical study of dynamic variable ordering heuristics for the CSP,” in Principles and Practice of Constraint Programming CP96, Lecture Notes in Computer Science, E. Freuder, Ed., vol. 1118, pp. 179–193, Springer, Berlin, Germany, 1996. View at: Google Scholar
  45. D. D. Lewis, “Naive (bayes) at forty: the independence assumption in information retrieval,” in Machine Learning: ECML-98, pp. 4–15, Springer, Berlin, Germany, 1998. View at: Publisher Site | Google Scholar
  46. H. Lourenço, O. Martin, and T. Stützle, “Iterated local search,” in Handbook of Metaheuristics, International Series in Operations Research & Management Science, F. Glover, G. Kochenberger, and F. S. Hillier, Eds., vol. 57, pp. 320–353, Springer, New York, NY, USA, 2003. View at: Google Scholar
  47. T. El-Ghazali, Metaheuristics: From Design to Implementation, Wiley Publishing, Hoboken, NJ, USA, 2009.
  48. F. T. Leighton, “A graph coloring algorithm for large scheduling problems,” Journal of Research of the National Bureau of Standards, vol. 84, no. 6, pp. 489–506, 1979. View at: Publisher Site | Google Scholar
  49. P. Augerat, J. M. Belenguer, E. Benavent et al., “Computational results with a branch and cut code for the capacitated vehicle routing problem,” 1995. View at: Google Scholar
  50. N. Christofides, “The vehicle routing problem,” in Combined Optimization, Wiley, New York, NY, USA, 1979. View at: Google Scholar
  51. B. L. Golden, E. A. Wasil, J. P. Kelly et al., “The impact of metaheuristics on solving the vehicle routing problem: algorithms, problem sets, and computational results,” in Fleet Management and Logistics, pp. 33–56, Springer, Berlin, Germany, 1998. View at: Google Scholar
  52. E. Uchoa, D. Pecin, A. Pessoa, M. Poggi, T. Vidal, and A. Subramanian, “New benchmark instances for the capacitated vehicle routing problem,” European Journal of Operational Research, vol. 257, no. 3, pp. 845–858, 2017. View at: Publisher Site | Google Scholar
  53. Z. Hanusz, J. Tarasinska, and W. Zielinski, “Shapiro-Wilk test with known mean,” REVSTAT-Statistical Journal, vol. 14, no. 1, pp. 89–100, 2016. View at: Google Scholar

Copyright © 2021 Lucero Ortiz-Aguilar et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Related articles

No related content is available yet for this article.
 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder
Views386
Downloads634
Citations

Related articles

No related content is available yet for this article.

Article of the Year Award: Outstanding research contributions of 2021, as selected by our Chief Editors. Read the winning articles.