Novel Global Harmony Search Algorithm for Least Absolute Deviation
The method of least absolute deviation (LAD) finds applications in many areas, due to its robustness compared to the least squares regression (LSR) method. LAD is robust in that it is resistant to outliers in the data. This may be helpful in studies where outliers may be ignored. Since LAD is nonsmooth optimization problem, this paper proposed a metaheuristics algorithm named novel global harmony search (NGHS) for solving. Numerical results show that the NGHS method has good convergence property and effective in solving LAD.
Least squares regression (LSR) method is one of the oldest and the most widely used statistical tools for linear models. Its theoretical properties have been extensively studied and are fully understood. Despite its many superior properties, the LSR estimate can be sensitive to outliers and, therefore, nonrobust . In order to overcome these problems, the researchers have recently investigated the alternative regression method which is least absolute deviation (LAD) method.
Least absolute deviation (LAD), also known as least absolute errors (LAE), least absolute value (LAV), or least absolute residual (LAR), or the norm problem, is a mathematical optimization technique similar to the LSR that attempts to find a function which closely approximates a set of data . In the simple case, the approximation function is a simple “trend line” in two-dimensional Cartesian coordinates . The method minimizes the sum of absolute errors (SAE) (the sum of the absolute values of the vertical “residuals” between points generated by the function and corresponding points in the data).
The method of LAD finds applications in many areas, due to its robustness compared to the LSR method. LAD is robust in that it is resistant to outliers in the data. This may be helpful in studies where outliers may be safely and effectively ignored. Despite its long history and many ground-breaking works, the LAD has not been explored in theory as well as in application to the extent as the LSR . This is largely because computing LAD estimates are more difficult than computing LSR estimates due to the fact that algorithmic method must be employed to calculate LAD estimates. Therefore, LAD estimation method is not popular as LSR method.
Over the past few years, a number of approaches have been developed for solving the LAD using classical mathematical programming methods. Since Charnes et al.  reduced the LAD method to a linear programming problem, the computational difficulty is now entirely overcome by the availability of computing power and the effectiveness of linear programming. A comprehensive summary of the subject can be found in Koenker and D’Orey . Large sample properties of the LAD estimates are obtained in Koenker and Bassett Jr. , Chen et al. , and Pollard . Due to these developments in theoretical and computational aspects, the LAD method has become increasingly popular. In particular, it has many applications in econometrics and biomedical studies; see Bassett and Koenker , Powell , Buchinsky , among many others.
Since LAD is a class of nondifferentiable optimization problem, this paper proposed a metaheuristics algorithm named novel global harmony search (NGHS)  for solving LAD model. The NGHS algorithm includes two important operations: position updating and genetic mutation with a small probability (). The former enables the worst harmony of harmony memory to move to the global best harmony rapidly in each iteration, and the latter can effectively maintain population diversity and prevent the NGHS from trapping into the local optimum.
The remaining sections of this chapter are organized as follows. In Section 2, LAD model is shown. In Sections 3 and 4, the classical HS and NGHS are described. Numerical results are presented and compared in Section 5. Lastly, Section 6 outlines our conclusions.
2. Least Absolute Deviations Model
The LAD method is a widely known alternative to the classical LSR method for statistical analysis of linear regression models. Instead of minimizing the sum of squared errors (SSE) in LSR, it minimizes the sum of absolute errors (SAE).
Suppose that the data set consists of points (data pairs) , where is an independent variable and is a dependent variable whose value is found by observation. We want to find a function such that .
To attain this goal, we suppose that the function is of a particular form containing some parameters which need to be determined. For instance, the simplest form would be linear: , where and are parameters whose values are not known but which we would like to estimate.
We now seek estimated values of the unknown parameters that minimize the SAE:
Let Then LAD problem is equivalent to following optimization problem: Meanwhile, then LSR problem is equivalent to following optimization problem For a given data set, the method of LAD may produce multiple solutions, whereas the method of LSR always produces only one solution (the regression line is unique).
3. Harmony Search Algorithm
Recently, a new class of metaheuristics, named harmony search (HS), has been developed. The HS algorithm proposed in  has been developed in an analogy with music improvisation process where musicians in an ensemble continue to polish their pitches in order to obtain better harmony. Jazz improvisation seeks to find musically pleasing harmony similar to the optimum design process which seeks to find optimum solution. The pitch of each musical instrument determines the aesthetic quality, just as the objective function value is determined by the set of values assigned to each decision variable. In addition, HS uses a stochastic random search instead of a gradient search.
The steps in the procedure of standard harmony search algorithm (HS) are as follows.
Step 1 (initialize the problem and algorithm parameters). The optimization problem is specified as follows: where is an objective function; is the set of each decision variable ; is the number of decision variables; is the set of the possible range of values for each decision variable; . The HS algorithm parameters are also specified in this step. These are the harmony memory size (HMS) or the number of solution vectors in the harmony memory; harmony memory considering rate (HMCR); pitch adjusting rate (PAR); and the number of improvisations (Tmax) or stopping criterion.
Step 2 (initialize the harmony memory). The HM matrix is filled with as many randomly generated solution vectors as the HMS
Step 3 (improvise a new harmony). Generate a new harmony is called “improvisation.” A new harmony vector, , is generated based on three rules: (1) memory consideration, (2) pitch adjustment, and (3) random selection. The procedure works as Algorthim 1.
is the th component of , and is the th component of the th candidate solution vector in HM. Here is uniform distribution random number in the region of , is an arbitrary distance bandwidth, and is uniform distribution random number in the region of .
Step 4 (update harmony memory). If the new harmony vector is better than the worst harmony in the HM, judged in terms of the objective function value, the new harmony is included in the HM and the existing worst harmony is excluded from the HM.
Step 5 (check stopping criterion). If the stopping criterion (maximum number of improvisations) is satisfied, computation is terminated. Otherwise, Steps 3 and 4 are repeated.
4. Novel Global Harmony Search Algorithm
Experiments with the classical HS algorithm over the benchmark problems show that the algorithm suffers from the problem of premature and/or false convergence and slow convergence especially over multimodal fitness landscape. To enrich the searching behavior and to avoid being trapped into local optimum, more improved HS algorithms were presented. Zou et al.  proposed an NGHS algorithm and it has been successfully used in reliability problems , and 0-1 knapsack problems .
The NGHS algorithm modifies the improvisation step of the HS such that the new harmony can mimic the global-best harmony in the HM. In Step 3 it works as Algorthim 2.
Here, ‘‘best’’ and ‘‘worst’’ are the indexes of the best and worst harmony in HM, respectively.
The NGHS proposed algorithm has strong global search ability in the early stage of optimization and has strong local search ability in the late stage of optimization. In the early stage of optimization, all solution vectors are sporadic in feasible space, so most is large, which is beneficial to the global search of the NGHS, while in the late stage of optimization, all nonbest solution vectors are inclined to move to the global-best solution vector, so most solution vectors are close to each other. In this case, most is small and most trust regions are narrow, which is beneficial to the local search of the NGHS.
In Step 4, the NGHS replaces the worst harmony in HM with the new harmony even if is worse than .
The NGHS and the HS are different in following. (i) In Step 1, harmony memory considering rate (HMCR), pitch adjusting rate (PAR), and adjusting step () are excluded from the NGHS, and genetic mutation probability () is included in the NGHS. (ii) The HS carries out mutation with the probability HMCR PAR and carries out random selection with the probability 1-HMCR, while the NGHS carries out genetic mutation with the probability . In fact, both operations are exactly the same, and they are used to keep the individual variety better, which can effectively prevent both algorithms from being trapped into the local optimum.
5. Computational Results
In this section we solve an LAD problem in order to illustrate the implementation and efficiency of the NGHS method. All the experiments were performed on MatlabR2009a system with Intel(R) Core(TM) GHz and 2 GB RAM.
The data pairs , are listed in Tables 1 and 2, where is an independent variable and is a dependent variable whose value is found by observation.
For data pairs listed in Tables 1 and 2, solved by LSR model , respectively, the results of data fitting are shown in Figure 1.
From Figure 1, LSR method is sensitive to outliers and, therefore, nonrobust. Its performance in terms of accuracy and statistical inferences may be compromised when the errors are large and heterogeneous.
Following we use LAD model to fit given data with outlier in Table 2. Since LAD is a kind of nondifferentiable optimization problem, we solved it by NGHS. In order to show the optimization capabilities of the NGHS, the results of classical HS, HSCH , and HSWB  are listed too. To make the comparison fair, the populations for all the competitor algorithms were initialized using the same random seeds. The HS-variants algorithm parameters were set the same parameters: harmony memory size HMS = 15, harmony memory consideration rate HMCR = 0.85, pitch adjusting rate PAR = 0.35, and Tmax = 400. In NGHS, we set . If using a random number, we set the random-number generator to the state of 0 so that the same data can be regenerated.
To judge the accuracy of different algorithms, 10 independent runs of each of the four algorithms were carried out and the best, the mean, and the worst fitness values and the standard deviation (Std) were recorded in Table 3.
Figure 2 shows the convergence and its boxplot figure of the best fitness in the population for the different algorithms. The values plotted for every generation are averaged over 10 independent runs. As can be seen, the NGHS algorithm is the best.
Moreover, the results of data fitting by NGHS algorithm are shown in Figure 3.
Unlike the LSR method, the LAD method is not sensitive to outliers and produces robust estimates.
The LAD method is a widely known alternative to the classical LSR method for statistical analysis of linear regression models. Since LAD is a class of nondifferentiable optimization problem, this paper given NGHS algorithm for solving. Unlike the LSR, the LAD is not sensitive to outliers and produces robust estimates. Moreover, NGHS algorithm is effective for solving LAD model. The future works include investigating the application of LAD or other algorithm for solving LAD.
Conflict of Interests
The author declares that there is no conflict of interests regarding the publication of this paper.
This work is supported by Hanzhong Administration of Science and Technology under Grant no. 2013hzzx-39.
S. P. Ellis, “Instability of least squares, least absolute deviation and least median of squares linear regression,” Statistical Science A, vol. 13, no. 4, pp. 337–350, 1998.View at: Publisher Site | Google Scholar | MathSciNet
M. Caner, “A note on least absolute deviation estimation of a threshold model,” Econometric Theory, vol. 18, no. 3, pp. 800–814, 2002.View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
S. Portnoy and R. Koenker, “The Gaussian hare and the Laplacian tortoise: computability of squared-error versus absolute-error estimators,” Statistical Science, vol. 12, no. 4, pp. 279–300, 1997.View at: Publisher Site | Google Scholar | MathSciNet
A. Charnes, W. W. Cooper, and R. O. Ferguson, “Optimal estimation of executive compensation by linear programming,” Management Science, vol. 1, no. 2, pp. 138–151, 1955.View at: Google Scholar | MathSciNet
R. Koenker and V. D'Orey, “Algorithm AS 229: computing regression quantiles,” Journal of the Royal Statistical Society C: Applied Statistics, vol. 36, no. 3, pp. 383–393, 1987.View at: Publisher Site | Google Scholar
R. Koenker and G. Bassett Jr., “Regression quantiles,” Econometrica, vol. 46, no. 1, pp. 33–50, 1978.View at: Publisher Site | Google Scholar | MathSciNet
X. R. Chen, Z. D. Bai, L. C. Zao, and Y. H. Wu, “Asymptotic normality of minimum -norm estimates in linear models,” Science in China, vol. 33, no. 11, pp. 1311–1328, 1990.View at: Google Scholar | MathSciNet
D. Pollard, “Asymptotics for least absolute deviation regression estimators,” Econometric Theory, vol. 7, no. 2, pp. 186–199, 1991.View at: Publisher Site | Google Scholar | MathSciNet
J. Bassett and R. Koenker, “Asymptotic theory of least absolute error regression,” Journal of the American Statistical Association, vol. 73, no. 363, pp. 618–622, 1978.View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
J. L. Powell, “Least absolute deviations estimation for the censored regression model,” Journal of Econometrics, vol. 25, no. 3, pp. 303–325, 1984.View at: Publisher Site | Google Scholar | Zentralblatt MATH | MathSciNet
M. Buchinsky, “Recent advances in quantile regression models: a practical guideline for empirical research,” Journal of Human Resources, vol. 33, no. 1, pp. 88–126, 1998.View at: Publisher Site | Google Scholar
D. X. Zou, L. Q. Gao, J. H. Wu, and S. Li, “Novel global harmony search algorithm for unconstrained problems,” Neurocomputing, vol. 73, no. 16–18, pp. 3308–3318, 2010.View at: Publisher Site | Google Scholar
K. S. Lee and Z. W. Geem, “A new meta-heuristic algorithm for continuous engineering optimization: harmony search theory and practice,” Computer Methods in Applied Mechanics and Engineering, vol. 194, no. 36–38, pp. 3902–3933, 2005.View at: Publisher Site | Google Scholar
D. Zou, L. Gao, J. Wu, S. Li, and Y. Li, “A novel global harmony search algorithm for reliability problems,” Computers and Industrial Engineering, vol. 58, no. 2, pp. 307–316, 2010.View at: Publisher Site | Google Scholar
D. Zou, L. Gao, S. Li, and J. Wu, “Solving 0-1 knapsack problem by a novel global harmony search algorithm,” Applied Soft Computing Journal, vol. 11, no. 2, pp. 1556–1564, 2011.View at: Publisher Site | Google Scholar
L. Yong, S. Liu, J. Zhang, and Q. Feng, “An improved harmony search algorithm for linear least squares,” Journal of Computational Information Systems, vol. 8, no. 10, pp. 4217–4224, 2012.View at: Google Scholar
L. Yong, S. Liu, S. Tuo, and K. Gao, “Improved harmony search algorithm with chaos for absolute value equation,” TELKOMNIKA, vol. 11, no. 4, pp. 835–844, 2013.View at: Google Scholar
L. Yong, S. Liu, S. Tuo et al., “Improved harmony search algorithm for absolute value equation,” Journal of Natural Science of Heilongjiang University, vol. 30, no. 3, pp. 321–327, 2013.View at: Google Scholar