Novel Global Harmony Search Algorithm for Least Absolute Deviation

Abstract

The method of least absolute deviation (LAD) finds applications in many areas, due to its robustness compared to the least squares regression (LSR) method. LAD is robust in that it is resistant to outliers in the data. This may be helpful in studies where outliers may be ignored. Since LAD is nonsmooth optimization problem, this paper proposed a metaheuristics algorithm named novel global harmony search (NGHS) for solving. Numerical results show that the NGHS method has good convergence property and effective in solving LAD.

1. Introduction

Least squares regression (LSR) method is one of the oldest and the most widely used statistical tools for linear models. Its theoretical properties have been extensively studied and are fully understood. Despite its many superior properties, the LSR estimate can be sensitive to outliers and, therefore, nonrobust [1]. In order to overcome these problems, the researchers have recently investigated the alternative regression method which is least absolute deviation (LAD) method.

Least absolute deviation (LAD), also known as least absolute errors (LAE), least absolute value (LAV), or least absolute residual (LAR), or the norm problem, is a mathematical optimization technique similar to the LSR that attempts to find a function which closely approximates a set of data . In the simple case, the approximation function is a simple “trend line” in two-dimensional Cartesian coordinates [2]. The method minimizes the sum of absolute errors (SAE) (the sum of the absolute values of the vertical “residuals” between points generated by the function and corresponding points in the data).

The method of LAD finds applications in many areas, due to its robustness compared to the LSR method. LAD is robust in that it is resistant to outliers in the data. This may be helpful in studies where outliers may be safely and effectively ignored. Despite its long history and many ground-breaking works, the LAD has not been explored in theory as well as in application to the extent as the LSR [3]. This is largely because computing LAD estimates are more difficult than computing LSR estimates due to the fact that algorithmic method must be employed to calculate LAD estimates. Therefore, LAD estimation method is not popular as LSR method.

Over the past few years, a number of approaches have been developed for solving the LAD using classical mathematical programming methods. Since Charnes et al. [4] reduced the LAD method to a linear programming problem, the computational difficulty is now entirely overcome by the availability of computing power and the effectiveness of linear programming. A comprehensive summary of the subject can be found in Koenker and D’Orey [5]. Large sample properties of the LAD estimates are obtained in Koenker and Bassett Jr. [6], Chen et al. [7], and Pollard [8]. Due to these developments in theoretical and computational aspects, the LAD method has become increasingly popular. In particular, it has many applications in econometrics and biomedical studies; see Bassett and Koenker [9], Powell [10], Buchinsky [11], among many others.

Since LAD is a class of nondifferentiable optimization problem, this paper proposed a metaheuristics algorithm named novel global harmony search (NGHS) [12] for solving LAD model. The NGHS algorithm includes two important operations: position updating and genetic mutation with a small probability (). The former enables the worst harmony of harmony memory to move to the global best harmony rapidly in each iteration, and the latter can effectively maintain population diversity and prevent the NGHS from trapping into the local optimum.

The remaining sections of this chapter are organized as follows. In Section 2, LAD model is shown. In Sections 3 and 4, the classical HS and NGHS are described. Numerical results are presented and compared in Section 5. Lastly, Section 6 outlines our conclusions.

2. Least Absolute Deviations Model

The LAD method is a widely known alternative to the classical LSR method for statistical analysis of linear regression models. Instead of minimizing the sum of squared errors (SSE) in LSR, it minimizes the sum of absolute errors (SAE).

Suppose that the data set consists of points (data pairs) , where is an independent variable and is a dependent variable whose value is found by observation. We want to find a function such that .

To attain this goal, we suppose that the function is of a particular form containing some parameters which need to be determined. For instance, the simplest form would be linear: , where and are parameters whose values are not known but which we would like to estimate.

We now seek estimated values of the unknown parameters that minimize the SAE:

Let Then LAD problem is equivalent to following optimization problem: Meanwhile, then LSR problem is equivalent to following optimization problem For a given data set, the method of LAD may produce multiple solutions, whereas the method of LSR always produces only one solution (the regression line is unique).

3. Harmony Search Algorithm

Recently, a new class of metaheuristics, named harmony search (HS), has been developed. The HS algorithm proposed in [13] has been developed in an analogy with music improvisation process where musicians in an ensemble continue to polish their pitches in order to obtain better harmony. Jazz improvisation seeks to find musically pleasing harmony similar to the optimum design process which seeks to find optimum solution. The pitch of each musical instrument determines the aesthetic quality, just as the objective function value is determined by the set of values assigned to each decision variable. In addition, HS uses a stochastic random search instead of a gradient search.

The steps in the procedure of standard harmony search algorithm (HS) are as follows.

Step 1 (initialize the problem and algorithm parameters). The optimization problem is specified as follows: where is an objective function; is the set of each decision variable ; is the number of decision variables; is the set of the possible range of values for each decision variable; . The HS algorithm parameters are also specified in this step. These are the harmony memory size (HMS) or the number of solution vectors in the harmony memory; harmony memory considering rate (HMCR); pitch adjusting rate (PAR); and the number of improvisations (Tmax) or stopping criterion.

Step 2 (initialize the harmony memory). The HM matrix is filled with as many randomly generated solution vectors as the HMS

Step 3 (improvise a new harmony). Generate a new harmony is called “improvisation.” A new harmony vector, , is generated based on three rules: (1) memory consideration, (2) pitch adjustment, and (3) random selection. The procedure works as Algorthim 1.
is the th component of , and is the th component of the th candidate solution vector in HM. Here is uniform distribution random number in the region of , is an arbitrary distance bandwidth, and is uniform distribution random number in the region of .

For each  𝑖∈[1,2,…,𝑁]  do    If  rand() < HMCR    𝑥′𝑖=𝑥𝑗𝑖(𝑗=1,2,…,HMS)   //memory consideration    If rand() < PAR     𝑥′𝑖=𝑥′𝑖+𝑟(−1,1)×𝑏𝑤     // pitch adjustment     𝑥′𝑖=min[max(𝑥′𝑖,𝑥𝑖𝐿),𝑥𝑖𝑈]  //truncation processing    End    Else     𝑥′𝑖=𝑥𝑖𝐿+rand()×(𝑥𝑖𝑈−𝑥𝑖𝐿)  //random selection    End End

Algorithm 1 

Generating a new harmony by classical HS algorithm.

Step 4 (update harmony memory). If the new harmony vector is better than the worst harmony in the HM, judged in terms of the objective function value, the new harmony is included in the HM and the existing worst harmony is excluded from the HM.

Step 5 (check stopping criterion). If the stopping criterion (maximum number of improvisations) is satisfied, computation is terminated. Otherwise, Steps 3 and 4 are repeated.

4. Novel Global Harmony Search Algorithm

Experiments with the classical HS algorithm over the benchmark problems show that the algorithm suffers from the problem of premature and/or false convergence and slow convergence especially over multimodal fitness landscape. To enrich the searching behavior and to avoid being trapped into local optimum, more improved HS algorithms were presented. Zou et al. [12] proposed an NGHS algorithm and it has been successfully used in reliability problems [14], and 0-1 knapsack problems [15].

The NGHS algorithm modifies the improvisation step of the HS such that the new harmony can mimic the global-best harmony in the HM. In Step 3 it works as Algorthim 2.

For each  𝑖∈[1,2,…,𝑁]  do  𝑥𝑠=2×𝑥best𝑖−𝑥worst𝑖  𝑥𝑠=min[max(𝑥𝑠,𝑥𝑖𝐿),𝑥𝑖𝑈]     //truncation processing  𝑥′𝑖=𝑥worst𝑖+rand()×(𝑥𝑠−𝑥worst𝑖)   //position updating  If  rand()<𝑝𝑚     𝑥′𝑖=𝑥𝑖𝐿+rand()×(𝑥𝑖𝑈−𝑥𝑖𝐿) //genetic mutation  End End

Algorithm 2 

Generating a new harmony by NGHS algorithm.

Here, ‘‘best’’ and ‘‘worst’’ are the indexes of the best and worst harmony in HM, respectively.

The NGHS proposed algorithm has strong global search ability in the early stage of optimization and has strong local search ability in the late stage of optimization. In the early stage of optimization, all solution vectors are sporadic in feasible space, so most is large, which is beneficial to the global search of the NGHS, while in the late stage of optimization, all nonbest solution vectors are inclined to move to the global-best solution vector, so most solution vectors are close to each other. In this case, most is small and most trust regions are narrow, which is beneficial to the local search of the NGHS.

In Step 4, the NGHS replaces the worst harmony in HM with the new harmony even if is worse than .

The NGHS and the HS are different in following. (i) In Step 1, harmony memory considering rate (HMCR), pitch adjusting rate (PAR), and adjusting step () are excluded from the NGHS, and genetic mutation probability () is included in the NGHS. (ii) The HS carries out mutation with the probability HMCR PAR and carries out random selection with the probability 1-HMCR, while the NGHS carries out genetic mutation with the probability . In fact, both operations are exactly the same, and they are used to keep the individual variety better, which can effectively prevent both algorithms from being trapped into the local optimum.

5. Computational Results

In this section we solve an LAD problem in order to illustrate the implementation and efficiency of the NGHS method. All the experiments were performed on MatlabR2009a system with Intel(R) Core(TM)  GHz and 2 GB RAM.

The data pairs , are listed in Tables 1 and 2, where is an independent variable and is a dependent variable whose value is found by observation.

Table 1 

Given data.

Table 2 

Given data with outlier.

For data pairs listed in Tables 1 and 2, solved by LSR model [16], respectively, the results of data fitting are shown in Figure 1.

Figure 1 

Data fitting results by LSR method. Here 𝑓1(𝑥)=−0.8940𝑥+18.6746 is fitting results of data in Table 1, and 𝑓2(𝑥)=−0.2288𝑥+6.4055 is fitting results of data in Table 2.

From Figure 1, LSR method is sensitive to outliers and, therefore, nonrobust. Its performance in terms of accuracy and statistical inferences may be compromised when the errors are large and heterogeneous.

Following we use LAD model to fit given data with outlier in Table 2. Since LAD is a kind of nondifferentiable optimization problem, we solved it by NGHS. In order to show the optimization capabilities of the NGHS, the results of classical HS, HSCH [17], and HSWB [18] are listed too. To make the comparison fair, the populations for all the competitor algorithms were initialized using the same random seeds. The HS-variants algorithm parameters were set the same parameters: harmony memory size HMS = 15, harmony memory consideration rate HMCR = 0.85, pitch adjusting rate PAR = 0.35, and Tmax = 400. In NGHS, we set . If using a random number, we set the random-number generator to the state of 0 so that the same data can be regenerated.

To judge the accuracy of different algorithms, 10 independent runs of each of the four algorithms were carried out and the best, the mean, and the worst fitness values and the standard deviation (Std) were recorded in Table 3.

Table 3 

The Statistical results for 10 runs on given data in Table 2.

Figure 2 shows the convergence and its boxplot figure of the best fitness in the population for the different algorithms. The values plotted for every generation are averaged over 10 independent runs. As can be seen, the NGHS algorithm is the best.

(a)

(b)

Figure 2 

The convergence and its boxplot of the best fitness for LAD.

Moreover, the results of data fitting by NGHS algorithm are shown in Figure 3.

Figure 3 

Data fitting results by LSR and LAD. Here 𝑓1(𝑥)=−0.8940𝑥+18.6746 is fitting results of data in Table 1 by LSR, 𝑓2(𝑥)=−0.2288𝑥+6.4055 is fitting results of data in Table 2 by LSR, and 𝑓3(𝑥)=−0.7306𝑥+15.5965 is fitting results of data in Table 2 by LAD.

Unlike the LSR method, the LAD method is not sensitive to outliers and produces robust estimates.

6. Conclusion

The LAD method is a widely known alternative to the classical LSR method for statistical analysis of linear regression models. Since LAD is a class of nondifferentiable optimization problem, this paper given NGHS algorithm for solving. Unlike the LSR, the LAD is not sensitive to outliers and produces robust estimates. Moreover, NGHS algorithm is effective for solving LAD model. The future works include investigating the application of LAD or other algorithm for solving LAD.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

This work is supported by Hanzhong Administration of Science and Technology under Grant no. 2013hzzx-39.