[Retracted] Active Learning Query Strategies for Linear Regression Based on Efficient Global Optimization

Zong, Tianxin; Li, Na; Zhang, Zhigang

doi:https://doi.org/10.1155/2022/2891463

Journal of Electrical and Computer Engineering

On this page

Abstract Introduction Related Work Results Conclusions Data Availability Conflicts of Interest Acknowledgments Supplementary Materials References Copyright Related Articles

Research Article Retraction

!

This article has been Retracted. To view the article details, please click the ‘Retraction’ tab above.

Special Issue

Big Data Analytics in Mobile Information Systems for Advanced Computing

View this Special Issue

Research Article | Open Access

Volume 2022 | Article ID 2891463 | https://doi.org/10.1155/2022/2891463

[Retracted] Active Learning Query Strategies for Linear Regression Based on Efficient Global Optimization

Tianxin Zong,¹Na Li,¹and Zhigang Zhang¹

Academic Editor: Xuefeng Shao

Received16 Apr 2022

Revised04 May 2022

Accepted11 May 2022

Published09 Jun 2022

Abstract

Active learning, a subfield of machine learning, can train a good model by selecting a minimum number of labeled samples. In many machine learning scenarios, needed information (such as the best value in unlabeled datasets) is acquired by prediction. When there is too little data in the training model, the prediction accuracy would obviously affect the accuracy of the results. To establish a high-performance regression model for a small dataset while accelerating the search for the best sample, a new active learning query strategy, EGO-ALR, that combines efficient global optimization (EGO) and active learning for regression (ALR) was proposed. It was found that the performance of EGO-ALR was significantly better than the original ALR query strategies in terms of the root mean square error (RMSE), correlation coefficient (CC), and opportunity cost (Oppo Cost). Specifically, EGO-ALR increased the Oppo Cost by an average of 25.27% when the RMSE and CC values were not more than 1.07% different from the original ALR. This study validated the efficiency and robustness of EGO-ALR approaches using 19 datasets from various domains and three distinct linear regression models (Ridge regression, Lasso, and Elastic network).

1. Introduction

Regression refers to estimating the value of a dependent variable (output) from one or more independent variables (characteristics). In a practical regression problem, some labeled samples (the independent variables and dependent variables are known) need to be trained by an appropriate approach to establish an accurate regression model. In general, the quality of performance of the trained model is proportional to the number of labeled samples. Data annotation is usually the biggest bottleneck in machine learning. Searching, managing, and labeling large amounts of data are often time-consuming and expensive to train a good model [1]. For example, in emotion estimation from speech signals, it is easy to record several speech utterances, but multiple assessors are needed to evaluate the emotion primitives [2]. In the problem of new alloy design, one can freely adjust the material composition within the range, but the synthesis and characterization of new materials should simultaneously consider the synthesis difficulty and cost of materials [3]. Similarly, in the research of video recommendation systems, users can simply upload videos, but few people manually annotate the metadata in detail; thus, costly annotation by experts is required, which will lead to a severe lack of text views and to a lack of training data for recommender systems [4].

To enable use of applications with missing labeled samples, investigators propose ALR [5]. ALR can sequentially select some of the most beneficial samples for labeling, so that the trained model gives the most accurate predictions for the remaining unlabeled samples. ALR is iterative: first, one builds an initial model from a small number of labeled training samples, and then by some selection strategies, the most valuable samples among the unlabeled samples are labeled and added to the training set for the next round of modeling. This process will iterate until stopping conditions are met, such as the maximum number of iterations, the maximum number of labeled samples, and the cross-validation accuracy of the model.

According to different query scenarios, investigators divide ALR into population-based ALR, flow-based ALR, and pool-based ALR [6]. The pool-based situation is considered in this study, in which a pool of unlabeled samples is given, and the goal is to select some training samples from the pool to improve a linear regression model.

When a machine learning regression model is established, people usually use this model to predict samples with unknown labels and then obtain the needed information from the prediction results. In material design problems, machine learning is frequently used to extrapolate to a vast unexplored search space to search for the best performing material [7], but the accuracy of predictions is closely related to the performance of the regression models. To guide the experiment to the ideal material quickly, Balachandran et al. combined active learning with experimental design and proposed an adaptive iterative design strategy [3] to accelerate the material discovery process. This strategy is used in the scenario of multiple material designs [8–11].

The adaptive strategy first defines a utility equation as the key to selecting the subsequent experimental sample. Then, the strategy predicts the most beneficial sample for experimental verification, and finally the strategy feeds the verified data to the machine learning model to improve the accuracy of predicting the best value. In this way, the samples with the best target performance are screened out with the least number of experiments. The utility functions commonly use the EGO algorithm [12] and the knowledge gradient algorithm [13].

ALR can only solve the problem using as few samples as possible to improve the ability of the machine learning model. However, training a high-precision regression model is not its ultimate goal in practical application. People prefer to predict the samples in an unknown space to get the best beyond the existing one. Although EGO can meet the needs of objective optimization, this strategy often sacrifices the precision of the model’s prediction while performing rapid optimization [3]. So, there is currently no query strategy that perfectly balances the prediction performance and optimization performance under small sample conditions.

It is easy to find that the principle of adaptive strategy is similar to ALR. Both strategies select the most beneficial samples from some approaches. If the utility function of global optimization is integrated with ALR, can we accelerate the progress to finding the best samples while building a model with high predictive performance? To address this question, this paper studied a class of ALR approaches based on optimization algorithms. The principal contributions are the following:(i)A brand-new AL query strategy, combining the EGO algorithm with the ALR approach, was proposed to optimally balance the needs of “exploitation” (aims at improving the predictive model) and “exploration” (aims at finding the best sample).(ii)The EGO-ALR inherits the usage of EGO: it can freely change the optimization direction, and its performance is stable regardless of finding the minimum or maximum value.(iii)Extensive experiments were carried out on three common linear regression models and 19 datasets from different application domains, demonstrating the effectiveness and robustness of EGO-ALR. It also shows that this query strategy can even outperform the original ALR in both prediction and optimization performance in some cases.

The remainder of this paper is organized as follows. Section 2 introduces the EGO algorithm and some existing pool-based ALR approaches. Section 3 elaborates on the combined framework of the proposed approach. Section 4 conducts extensive experiments on 19 datasets, elucidating the experimental results and superiority performance of our approach. Finally, the conclusion and future work are given in Section 5.

2.1. Existing Pool-Based ALR Query Strategies

The existing pool-based ALR approaches can be classified into two scenarios: supervised and unsupervised. Most existing ALR approaches are supervised; these approaches need some ground-truth labels to guide the sample selection. Unsupervised ALR does not require any label information when selecting samples. Next, several commonly used supervised and unsupervised ALR approaches are introduced below.

2.1.1. Supervised ALR Query Strategies

Query by committee (QBC) [14] is widely used in different fields [15–19]. Assuming that N is the number of samples in the dataset, QBC first randomly selects and labels K0 samples, then establishes a committee of l learners from the existing training set (usually by bootstrapping), and predicts the samples in the unlabeled pool. QBC will select the samples from the pool on which the committee disagrees the most to label, that is:where and is the prediction result of the ith model built by bootstrap for the nth unlabeled sample x_n. QBC selects the sample with the largest to label.

Expected model change (EMCM) [20] is an ALR approach for regression and classification [21–23]; EMCM has a variety of algorithms [24–26]. Expected model change for linear regression is considered in this report. Expected model change first randomly selects and labels K₀ samples to train a linear regression model. The prediction result of the model for the nth unlabeled sample is set as . Then, EMCM uses bootstrap to build l linear regression models. In each sequential iteration, the labeled samples are those that change the linear regression parameters the most, that is:

EMCM labels the sample with the largest (x_n).

2.1.2. Unsupervised ALR Query Strategies

Yu and Kim proposed an unsupervised ALR approach based on greedy sampling [27], which is also applied to image signal processing [28]. Greedy sampling initially labels at least one sample, but Yu and Kim do not define the first sample. Therefore, the study used an improved method, GSx, proposed by Wu et al. [29]. The idea of GSx is to take the sample closest to the centroid in the pool as the first labeled sample and then select a sample in a greedy way such that it is farthest from all the selected samples at each sequential iteration:where x_m is the labeled sample. GSx first calculates the distance between x_n and x_m of all unlabeled samples, then it computes the minimum distance from x_n to x_m:and selects the sample with the largest d_n.

Representation-diversity (RD) proposed by Wu is also unsupervised and can be used for linear regression [30], and RD derived some excellent unsupervised ALRs such as IRD [31] and iRDM [1]. It performs k-means clustering (k = K₀) and selects the first K₀ samples closest to the centroid from each cluster. When selecting the K₀ + 1th sample, RD performs k-means clustering (k = K₀ + 1) on all samples in the pool, identifies the largest cluster that does not contain the labeled samples, and selects the one closest to the centroid as the K₀ + 1th sample. The basic RD algorithm can also be combined with other supervised ALR approaches for better performance [30]. For example, RD-EMCM combines RD and EMCM.

The ALR approach improves only the prediction performance as the criterion for selecting samples. Due to multiple limitations such as too few labeled samples, the complexity of the dataset distribution, and the defects of the algorithm itself, it is difficult for even the ALR to achieve the required prediction accuracy in only a few iterations. The results obtained by such a model have many deviations and cannot be used as a reference.

2.2. Efficient Global Optimization

EGO [12] is an algorithm with many related extensions to different types of research [32, 33]. We first introduce the expected improvement [34] before introducing EGO. Let f ^∗ = min (y₁, …, y_m) be the current best value in the training set. Before labeling x_n, its value y_n is uncertain. The uncertainty at y_n is modeled as the realization of a normally distributed random variable Y with mean and standard deviation determined by bootstrap. If the normal density function with the mean and standard deviation is plotted at x_n, y_n has a certain probability to be better than f ^∗. Expected improvement (EI) weighs all possible improvements by the associated density value at the point. Formally, the improvement at x_n is I = max (f∗-Y, 0). Because Y is a random variable, this expression is also a random variable. Simply take the expected value to obtain the EI:

To compute this expectation, the notations μ_n and σ_n are introduced to denote the expectation and standard deviation at x_n. Y is normal (μ_n, ). By performing some integrals by parts on the right side of equation (5), one can expand as

In the above equation, ϕ (·) and Φ (·) are the standard normal density and distribution function. EGO will select the sample with the largest E[I(x_n)].

The EGO considers both the predicted (uncertain) and optimized (best) value, but the prediction error of the EGO-constructed model is still high [3]. The advantage of EGO is that in the case of large regression error, the approach can also be more effective than random selection or direct selection of the best value of the prediction. Because of this characteristic, EGO is used mostly in data-driven material design applications, which require selecting samples with better performance by a small number of iterations.

3. ALR Query Strategies Integrated with EGO

ALR is a subfield of machine learning that can train a good model by selecting a minimum number of labeled samples, and EGO is a global optimization algorithm that can quickly find the best sample. This study proposes a query strategy that integrates EGO with ALR, called EGO-ALR. By combining the advantages of the EGO and the ALR during sample selection, the method can accelerate the optimization process while maintaining prediction quality, which effectively reduces the influence of model performance on the outcome.

The complete framework of the active learning method using EGO-ALR query strategy is shown in Figure 1. First, resample the training set by bootstrap for l times. Second, train the resampled set into a regression model through machine learning. Then, predict all samples in the pool and use the EGO-ALR query strategy to calculate the information of each unlabeled sample. Finally, select and mark the most informative sample and add it to the training set.

Due to the different principles of supervised and unsupervised ALR query strategies, this paper needs to discuss the different approaches of EGO combined with the two kinds of ALR query strategies and point out some special changes.

3.1. Combination of Supervised ALR and EGO

The pseudocode of the supervised ALR combined with EGO is given in Algorithm 1. Pool U firstly consists of N unlabeled samples and 0 labeled samples. Set K₀ as the number of samples in the initial training set. Because it is combined with the supervised ALR approach, all samples in the initial training set will be randomly selected and labeled. Assuming that the first K samples (K ≥ K₀) have been labeled, for the remaining N-K unlabeled samples, EGO-ALR first computes separately the “information” in both EGO and supervised ALR.

The “information” is defined as a measure of how valuable a sample is to be labeled, for example, used in QBC, (x_n) used in EMCM, and E[I(x_n)] used in EGO. Note that the “information” between each approach may have significantly different dimensions, and a larger scale may dominate the other “information.” Thus, EGO-ALR normalizes the “information” by min-max normalization and then adds it after weighting by the parameter c to reduce the sensitivity of the formula to scale. Here is an example of the “information” after the combination of EGO and QBC:where c is the adjustable weight and ∗ represents the normalized value. For labeling, EGO-ALR selects the sample with the largest T_n.

Because the value of parameter c is the most effective way to balance the prediction performance of ALR and the optimization performance of EGO, the value has a crucial influence on the sampling results. Generally, the larger the c value, the closer the sample selection to ALR, and model prediction performance increases while optimization performance (find the best value) decreases; the smaller the c value, the closer the sample selection to EGO, and model prediction performance decreases while optimization performance increases. The effect of different parameters c on the results is explained in Section 4.7.

(1)	Input: x_n, a pool of N unlabeled samples; K, maximum number of samples to label;
(2)	c, weighting parameters.
(3)	Output: regression model f(x)
(4)	Randomly select and label K₀ samples;
(5)	Construct the initial regression model f(x) with K₀ samples;
(6)	for m = K₀ + 1, …, K do
(7)	Build L regression models using bootstrap from the training set
(8)	for n = m, …, K do
(9)	EGO-QBC: compute in (1) and E[I(x_n)] in (6);
(10)	min-max normalization of and E[I(x_n)], marked as and
(11)	Compute
(12)	EGO-EMCM: compute (x_n) in (2) and E[I(x_n)] in (6);
(13)	min-max normalization of (x_n) and E[I(x_n)], marked as (x_n)^∗ and
(14)	Compute
(15)	end
(16)	Label the sample with the largest T_n and add it to the training set.
(17)	end
(18)	Update the regression model f(x) with the labeled K samples.

3.2. Combination of Unsupervised ALR and EGO

The combined query strategy of unsupervised ALR is similar to supervised ALR, with the difference in the initial training samples. “Unsupervised” means the selection of samples is independent of the label information. When the pools are all unlabeled samples, unsupervised ALR can still select samples for labeling by corresponding strategies. Algorithm 2 gives the pseudocode. The combination of GSx and EGO uses the method described in Section 3.1. The combination of RD and EGO adopts the method described by Wu [30], that is, it selects RD to initialize and uses EGO to select samples in the largest cluster lacking labeled samples.

(1)	Input: x_n, a pool of N unlabeled samples; K, maximum number of samples to label;
(2)	c, weighting parameters.
(3)	Output: regression model f(x)
(4)	Select and label the initial K₀ samples with the GSx (or RD) algorithm;
(5)	Construct the initial regression model f(x) with K₀ samples;
(6)	for m = K₀ + 1, …, K do
(7)	Build L regression models using bootstrap from the training set
(8)	for n = m, …, K do
(9)	EGO-GSx: compute d_n in (4) and E[I(x_n)] in (6);
(10)	min-max normalization of d_n and E[I(x_n)], marked as and
(11)	Compute
(12)	RD-EGO: perform k-means (k = n) clustering on all samples in the pool;
(13)	Identify the largest cluster that does not contain labeled samples
(14)	Compute E[I(x_n)] in (6) for the samples in the cluster
(15)	end
(16)	Label the sample with the largest T_n (or E[I(x_n)]) and add it to the training set.
(17)	end
(18)	Update the regression model f(x) with the labeled K samples.

In summary, EGO-ALR randomly (or using unsupervised ALR) selects the first K₀ samples to build an initial regression model, and, in each subsequent iteration, EGO-ALR chooses the sample with the largest T_n (or E[I(x_n)]) to achieve the combination and balance of “exploitation” and “exploration.”

4. Results

This section conducted experiments on 19 datasets and three linear regression models to establish the performance of the proposed EGO-ALR. The experimental device was a personal computer, and the programming language was MATLAB R2018b.

4.1. Data Sources

A total of 19 datasets were used in the experiment. Sixteen datasets were from the UCI Machine Learning Library and three were from the CMU StatLib Datasets Archive. These sources have been used in many ALR studies [1, 20, 23, 27, 29–31]. Table 1 summarizes the datasets. Before the experiment, all datasets were removed of samples with missing features, special characters, and garbled characters. Two datasets, AutoMPG and CPS, contained some categorical features, which needed to be converted into numerical features by one-hot coding before the experiment (this conversion increased the number of features).

In addition, there are three datasets from the field of materials collected from the literature: HEA [10], Direct [11], and Indirect [11]. Note that the features of the HEA dataset were calculated from the data and feature formula provided by the literature. Before the experiment, each dimension of the feature space was normalized by Z-score, so that the mean of the feature dimension was zero and the standard deviation was one.

4.2. Comparison Algorithm

The study compared the performance of 10 different approaches as follows:(1)Base line, BL, which randomly selects all samples for labeling.(2)EGO, which is introduced in Section 2.(3)EMCM: supervised ALR, which is introduced in Section 2.1.(4)EGO-EMCM (c = 2): the combination of EGO and EMCM, which is introduced in Section 3.1.(5)QBC: supervised ALR, which is introduced in Section 2.1.(6)EGO-QBC (c = 2): the combination of EGO and QBC, which is introduced in Section 3.1.(7)GSx: unsupervised ALR, which is introduced in Section 2.2.(8)EGO-GSx (c = 2): the combination of EGO and GSx, which is introduced in Section 3.2.(9)RD: unsupervised ALR, which is introduced in Section 2.2.(10)RD-EGO: the combination of EGO and RD, which is introduced in Section 3.2.

4.3. Evaluation Process

For each dataset, first randomly select 50% of the total samples as the training pool U and the remaining 50% as the test set T: U (50%) +T (50%). Because the benefits of the ALR method are reflected in modeling with a small number of samples, each approach selected K∈[5, 50] sample from the training set U. The entire process was repeated 100 times to eliminate the effect of randomness on the results.

After each iteration of each approach, RMSE and CC were computed as measures of prediction performance. To measure the ability to find the best value of different approaches, Oppo Cost was also used in the evaluation. Oppo Cost was defined as the modulus difference between the current best and the overall best [3]. Powell and Ryzhov [35] also used Oppo Cost to compare the performance of knowledge gradient and EGO algorithms. To horizontally compare different approaches on different datasets, the Oppo Cost in this paper specifically refers to the normalized Oppo Cost.

The formulas of the three evaluation indicators are as follows:where y_i is the actual value of the test set sample, is the predicted value of the test set sample, and n is the number of samples in the test set; i = 1, 2, …, n. μ^∗ is the best-so-far and μ^∗∗ is the overall best value in the training pool. y_max and y_min are the maximum and minimum values in the training pool. Note that RMSE and CC evaluate the ability of the regression model to predict the samples in the test set, whereas Oppo Cost evaluates the ability to find the samples with the best performance during the iteration. Thus, (10) is calculated for the training pool, not the test set.

Note that CC was not directly optimized in the objective function of these approaches [1, 29, 30]. Generally, a regression model with a CC close to 1 should have a smaller RMSE, but there is no guarantee (see the experimental results below for details). Thus, the CC can be viewed only as a secondary measure of prediction performance for reference.

For each approach, three regularized linear regression models were used for training:(i)Ridge regression, Ridge [36]: regularization coefficient λ = 0.1.(ii)Lasso regression, Lasso [37]: regularization coefficient λ = 0.1.(iii)Elastic network, Enet [38]: regularization coefficient λ = 0.1, penalty item mixing parameter α = 0.5.

The regularized regression was chosen over ordinary linear regression because the number of labeled samples was too small. Thus, the model, which regularized the coefficients, usually achieved better performance compared with the ordinary linear regression model.

4.4. Experimental Result on Ridge

Figure 2 shows the average RMSE, CC, and Oppo Cost of different optimization directions for 10 methods with 19 datasets when using Ridge as the regression model.(1)The performance of all ten approaches improved as the value of K increased (smaller RMSE, Oppo Cost, and larger CC), which was intuitive. However, because of the small number of samples in the early stage, there were some fluctuations in the result. For example, when K ∈ [5, 10], the RMSE and CC results on 15 of the 19 datasets showed unexpected changes (larger RMSE and smaller CC when K increases). Of course, this problem was lessened after K continued to increase.(2)Intuitively, the prediction performance of the ten approaches was better than BL in most datasets, which suggested that the samples selected by the strategy could indeed improve the performance of the regression model.(3)Most algorithms with better optimization performance are related to EGO. EGO-ALR approaches in different optimization directions all had smaller Oppo Costs compared with ALR approaches. EGO achieved the smallest Oppo Costs on 15 of the 19 datasets (the remaining four datasets had the second smallest Oppo Cost). The above shows that the EGO and the approaches combined with EGO were the best sample selection approaches for optimization, no matter finding the maximum or the minimum.

This study additionally computed the area under the curve (AUC) of the mean RMSEs, CCs, and Oppo Costs for the Ridge regression model, denoted as AUC-RMSE, AUC-CC, and AUC-OPPO, respectively, to compare the prediction and optimization performance more concretely between the approaches (Figure 3). Because the AUCs from different datasets varied greatly, the AUC results were normalized to the AUC of BL; thus, the AUC of BL was always 1.

We made the following observations:(1)On average, GSx had the largest AUC-CC (1.0857) and the smallest AUC-RMSE (0.8384) for most datasets. The prediction performance of QBC (AUC-CC = 1.0648, AUC-RMSE = 0.8831) was slightly better than EMCM (AUC-CC = 1.0588, AUC-RMSE = 0.8897), and both were better than RD (AUC-CC = 1.0224, AUC-RMSE = 0.9314). The performance of EGO (AUC-CC = 1.0297, AUC-RMSE = 0.9691) was only better than BL.(2)For most datasets, EGO-EMCM, EGO-QBE, and EGO-GSx had similar prediction performance relative to their original ALR. Specifically, the maximum absolute value of AUC-CC between the three combined approaches and their original ALR was 0.176, and the average was 0.007. The maximum absolute value of the AUC-RMSE difference was 0.111, and the average was 0.009. The results of RD-EGO were peculiar; its average AUC-CC was 1.0271 and greater than RD in 13 of the 19 datasets. Meanwhile, the average AUC-RMSE of RD-EGO was 0.9143 and smaller than RD in 13 datasets of the 19 datasets. The prediction performance of RD-EGO was better than the RD on average, which was consistent with the description of the performance of RD combined with other approaches reported by Wu [30].(3)The optimization results of RD (average 1.1638) were the worst for 17 of the 19 datasets and the second lowest on the other two datasets. The optimization performance of QBC (average 0.6542) was also slightly better than EMCM (average 0.6625). GSx (average 0.5803) was the best among the four ALR approaches, whereas EGO (average 0.43957), as a global optimization algorithm, had better performance compared with all approaches.(4)EGO-EMCM, EGO-QBC, and RD-EGO had significantly smaller AUC-OPPO than their original ALR in all datasets. EGO-GSx had smaller AUC-OPPO (average 0.4829) than GSx for 17 of 19 datasets. Of the remaining two datasets, the AUC-OPPO difference between EGO-GSx and GSx was at most 0.067. Generally, the optimization performance of the four EGO-ALRs was always better than all ALRs. EGO-GSx, as the combination approach of EGO and GSx, had better optimization performance than the other three EGO-ALRs. The optimization performance of the remaining three approaches, ranked from the best to the worst, was RD-EGO, EGO-QBC, and EGO-EMCM.

In summary, the rank of the prediction performance of the 10 approaches could be sorted as follows: GSx ≈ EGO-GSx > QBC ≈ EGO-QBC > EMCM ≈ EGO-EMCM > RD-EGO > RD > EGO > BL. The optimization performance ranking was EGO > EGO-GSx > RD-EGO > GSx > EGO-QBC > EGO-EMCM > QBC > EMCM > BL > RD. The rank confirms that our proposed method, whether combined with supervised ALR or unsupervised ALR, exhibited strong advantages in improving the optimization performance without significant loss of prediction performance.

The measurement standard of the algorithm is not only accuracy but also stability. In the case of similar algorithm performance, the more stable algorithm is usually chosen. Table 2 shows the percent improvement of the AUCs of the mean RMSEs and CCs over BL. Ridge was the regression model.

As seen from Table 2, the improvement of RMSE and CC of all ALR approaches was better than EGO (RMSE = 3.09%, CC = 1.81%). According to the results of RMSE, GSx (mean = 16.17%, std = 15.27%) had the largest and most stable improvement compared with BL, followed by EGO-GSx. CC showed that the improvement of GSx (mean = 7.65%, std = 2.14%) was the largest, the improvement of QBC (mean = 4.08%, std = 3.44%) was the most stable, and the improvement of the standard deviation of RD (−1.81%) and RD-EGO (−1.88%) was negative, which indicated that these two approaches were very unstable. EGO-ALR and its original ALRs had a similar improvement in CC and RMSE, and the difference in improvement was less than 1%.

EGO had the largest and most stable improvement in Oppo Cost compared to BL, while EGO-ALRs had a larger and more stable improvement than the four ALR approaches. These results correspond to the ranking of the performance.

4.5. Experimental Results on Lasso and Enet

All the foregoing experiments were repeated with Lasso and Enet models. The conclusions were similar to Section 4.4. For additional results, see Figures S1 and S2 and Tables S1 and S2 in the Supplementary Material. Compared with Ridge, the optimization performance of the 10 approaches was improved significantly with Lasso and Enet, and the standard deviation of RMSE was improved most obviously with Lasso.

To quantify the performance improvement of the four EGO-ALRs compared with their original ALRs, this study computed their percent improvements with the three regression models (Table 3). The lowest promotion percent on RMSE was −1.07%, and the lowest on CC was −1.99%. The lowest on Oppo Cost was 10.92%, and the highest was 55.45%. Regardless of the regression model, the percent improvement of EGO-ALRs in RMSE and CC was not less than −2%, but the percent improvement of Oppo Cost was more than 10%.

RD-EGO had the most significant improvement over RD; the Oppo Cost increased by 55.45% and the RMSE increased by more than 1% on both Ridge and Lasso. The improvement in the Oppo Cost of EGO-EMCM was second only to RD-EGO, and the CC of EGO-EMCM was also positive on the Lasso and Enet models.

4.6. Statistical Analysis

This section established the test groups that compared EGO-ALR with its original ALR, EGO, and BL to see if the differences in Oppo Cost between EGO-ALR and other approaches were statistically significant (EGO-ALR was used as the control approach). There were four EGO-ALRs in our work, so there were four test groups.

First, the Friedman test was performed on 19 datasets, and the calculated statistic Ff (Table 4) was proposed by Iman and Davenport [39]. All calculated Ff values were always greater than the critical value F (3,54) = 2.78, which suggested that, regardless of which regression model was used, there were statistical differences among these approaches in each group.

After that, the post hoc test was performed to compare methods. The power of the post hoc test is greater when all methods are compared only to the control method and not between each other [40]. Thus, we used the Bonferroni–Dunn test as post hoc test. At q = 0.1, the critical difference (CD) for comparing four approaches on 19 datasets was 0.8880, and the visualization of the post hoc test is shown in Figure 4.

(a)

(b)

(c)

Irrespective of the regression model, the opportunity cost of EGO-ALR was significantly better than that of the original ALR and BL. In addition, there was no significant difference in the average ranking between EGO-ALRs and EGO except for EGO-QBC on Lasso and Enet models. Thus, it is concluded that the performance improvement of the original ALR and BL by the EGO-ALR was statistically significant. However, the improvement was not significantly different from that of EGO.

4.7. Parameter c Sensitivity

EGO-ALR in Section 3.1 has a parameter c, the weighted value of “information.” This section investigated the effect of c on the performance of the EGO-ALR. Figure 5 shows the normalized AUCs of EGO-ALR on Ridge, Lasso, and Enet when the parameter c ∈ [0, 5]. The corresponding results of EGO are also marked on the figure for comparison. Note that EGO-ALR is equivalent to the EGO when c = 0.

The performance of EGO-ALR improved as c increased, and performance converged after c = 3. In general, the result of c = 1 was closer to EGO; too much emphasis on optimization performance leads to a large loss of prediction performance. So, c = 1 is not a recommended value. c > 2 can maximize the optimization performance without excessive loss of prediction performance. When c > 2, the AUC-CC results can even outperform ALR (c ≥ 2 on the Lasso model, c ≥ 3 on the Enet model). The larger the c, the stronger the model prediction performance; the smaller the c, the stronger the model optimization performance. This result is in line with the respective sample selection characteristics of EGO and ALR. To find the balance between predicting and optimizing that makes the choice from the EGO-ALR algorithm more meaningful, c = 2 was chosen as the parameter of EGO-ALR.

4.8. Visualization of Sample Selection

This section explains the selection behavior of different approaches by the visualization of sample selection, to better visualize the advantages of EGO-ALR.

Taking the results of modeling a typical dataset (HEA) using Ridge regression as an example, the study compared EGO, EMCM, EGO-EMCM, GSx, and EGO-GSx, including supervised ALR and unsupervised ALR. Figure 6 shows the visualization of the sample selection results after iterating to the 20th round (the number of labeled samples is 25). The first 24 samples selected by an approach are marked in blue. The samples selected in the 20th round are marked in red. The black dotted line represents that the predicted value was equal to the actual value, and the red dotted line is the optimization goal of this experiment, that is, the minimum of Y.

Driven by the global optimization strategy, EGO collected samples with low Y and hardly selected samples with high Y, which caused the overall sampling of EGO to be biased. Most of the unlabeled samples are above the black dotted line, which indicates that the predicted value of unlabeled samples was significantly lower than the actual value.

EMCM-selected samples were distributed uniformly in the entire space. However, because the early sampling of EMCM was random and the subsequent sampling was small, the prediction of unlabeled samples in this experiment was also lower than the actual value. This situation would be improved after increasing the number of samples.

GSx-selected samples were more uniform than EMCM and EGO samples, so the prediction results were significantly better. This selection strategy caused it to seldom focus on a cluster for sampling, and it was difficult to find the best value.

The sampling of EGO-EMCM and EGO-GSx followed the original ALR while favoring lower Y clusters. EGO-EMCM and EGO-GSx not only ensured the prediction performance but also had more opportunities to select the best sample. It is further confirmed that the EGO-based ALR approach selects more reasonable samples than the original ALR, which results in better regression performances.

5. Conclusions

This study presents the EGO-ALR query strategy, which combines the ALR and EGO via weighted addition of normalization “information.” EGO-ALR combines the benefits of the two original approaches, speeding up the process of optimizing samples while also establishing a high-precision regression model. EGO-ALR circumvents the complexities of sample labeling and the impact of model performance on the accuracy of subsequent results. Furthermore, depending on the demand, EGO-ALR can vary the search direction of the ideal value. The study used multiple ALR approaches and conducted extensive experiments with 19 datasets in different domains. The performance of the EGO-ALR was significantly better than the original ALR as evaluated by RMSE, CC, and opportunity cost. Specifically, EGO-ALR increased the opportunity cost by an average of 25.27% when the RMSE and CC values were not more than 1.07% different from the original ALR. Whether combined with supervised or unsupervised ALR, EGO-ALR had strong adaptability. In addition, the EGO-ALR evaluation results on Ridge are similar to those on Lasso and Enet regression models, demonstrating the stability of this approach in the linear regression model. To make the results of EGO-ALR meaningful, the value range of the parameter c≥ 2 is recommended.

As one of the future steps, EGO or more optimization algorithms can be combined with ALR approaches not mentioned in this report. Alternatively, the method of combined query strategy can be extended to a nonlinear regression model or classification problem. The single objective optimization problem in this paper can also be extended to multiobjective optimization problems.

Data Availability

The experimental data used to support the findings of the study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This study was funded by the University of Science and Technology Beijing (00012259). The authors thank AiMi Academic Services (http://www.aimieditor.com) for the English language editing and review services.

Supplementary Materials

Supplementary materials are the active learning experimental results of Lasso and Enet regression, which contains the normalized AUCs of all models on 19 datasets, and the AUCs percent improvements of each model over BL. These experiments are consistent with the experiments used Ridge regression in Figure 3 and Table 2 in the article. (Supplementary Materials)

References

Z. Liu, X. Jiang, H. Luo, W. Fang, J. Liu, and D. Wu, “Pool-based unsupervised active learning for regression using iterative representativeness-diversity maximization (IRDM),” Pattern Recognition Letters, vol. 142, pp. 11–19, 2021.
View at: Publisher Site | Google Scholar
M. Grimm, K. Kroschel, and S. Narayanan, “The Vera Am Mittag German audio-visual emotional speech database,” in Proceedings of the 2008 IEEE International Conference on Multimedia and Expo, pp. 865–868, Hannover, German, April 2008.
View at: Publisher Site | Google Scholar
P. V. Balachandran, D. Xue, J. Theiler, J. Hogden, and T. Lookman, “Adaptive strategies for materials design using uncertainties,” Scientific Reports, vol. 6, no. 1, pp. 19660–19669, 2016.
View at: Publisher Site | Google Scholar
J. J. Cai, J. Tang, Q. G. Chen, Y. Hu, X. Wang, and S. J. Huang, “Multi-view active learning for video recommendation,” in Proceedings of the International Joint Conference on Artificial Intelligence, pp. 2053–2059, Macao. P.R., China, August 2019.
View at: Google Scholar
B. Settles, “Active learning literature survey,” Science, vol. 10, pp. 237–304, 1995.
View at: Google Scholar
P. Kumar and A. Gupta, “Active learning query strategies for classification, regression, and clustering: a survey,” Journal of Computer Science and Technology, vol. 35, no. 4, pp. 913–945, 2020.
View at: Publisher Site | Google Scholar
T. Lookman, P. V. Balachandran, D. Xue, and R. Yuan, “Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design,” Npj Computational Materials, vol. 5, no. 1, pp. 21–17, 2019.
View at: Publisher Site | Google Scholar
D. Xue, P. V. Balachandran, J. Hogden, J. Theiler, D. Xue, and T. Lookman, “Accelerated search for materials with targeted properties by adaptive design,” Nature Communications, vol. 7, no. 1, pp. 11241–11249, 2016.
View at: Publisher Site | Google Scholar
R. Yuan, Z. Liu, P. V. Balachandran et al., “Accelerated discovery of large electrostrains in BaTiO 3 ‐based piezoelectrics using active learning,” Advanced Materials, vol. 30, no. 7, Article ID 1702884, 2018.
View at: Publisher Site | Google Scholar
C. Wen, Y. Zhang, C. Wang et al., “Machine learning assisted design of high entropy alloys with desired property,” Acta Materialia, vol. 170, pp. 109–117, 2019.
View at: Publisher Site | Google Scholar
M. Su, R. Grimes, S. Garg, D. Xue, and P. V. Balachandran, “Machine-learning-enabled prediction of adiabatic temperature change in lead-free BaTiO3-based electrocaloric c,” ACS Applied Materials & Interfaces, vol. 13, no. 45, pp. 53475–53484, 2021.
View at: Publisher Site | Google Scholar
D. R. Jones, M. Schonlau, and W. J. Welch, “Efficient global optimization of expensive black-box functions,” Journal of Global Optimization, vol. 13, no. 4, pp. 455–492, 1998.
View at: Publisher Site | Google Scholar
P. Frazier, W. Powell, and S. Dayanik, “The Knowledge-Gradient Policy for correlated normal beliefs,” INFORMS Journal on Computing, vol. 21, no. 4, pp. 599–613, 2009.
View at: Publisher Site | Google Scholar
T. RayChaudhuri and L. G. Hamey, “Minimisation of data collection by active learning,” in Proceedings of the IEEE International Conference on Neural Networks, pp. 1338–1341, Perth, WA, Australia, December 1995.
View at: Publisher Site | Google Scholar
Y. Freund, H. S. Seung, E. Shamir, and N. Tishby, “Selective sampling using the query by committee algorithm,” Machine Learning, vol. 28, no. 2/3, pp. 133–168, 1997.
View at: Publisher Site | Google Scholar
S. Buus, S. L. Lauemøller, P. Worning et al., “Sensitive quantitative predictions of peptide-MHC binding by a 'Query by Committee' artificial neural network approach,” Tissue Antigens, vol. 62, no. 5, pp. 378–384, 2003.
View at: Publisher Site | Google Scholar
R. Burbidge, J. J. Rowland, and R. D. King, “Active learning for regression based on query by committee,” in Proceedings of the International Conference on Intelligent Data Engineering and Automated Learning, pp. 209–218, Berlin, UK, December 2007.
View at: Publisher Site | Google Scholar
B. Demir and L. Bruzzone, “A multiple criteria active learning method for support vector regression,” Pattern Recognition, vol. 47, no. 7, pp. 2558–2567, 2014.
View at: Publisher Site | Google Scholar
J. Vandoni, E. Aldea, and S. Le Hégarat-Mascle, “Evidential query-by-committee active learning for pedestrian detection in high-density crowds,” International Journal of Approximate Reasoning, vol. 104, pp. 166–184, 2019.
View at: Publisher Site | Google Scholar
W. Cai, Y. Zhang, and J. Zhou, “Maximizing expected model change for active learning in regression,” in Proceedings of the IEEE 13th International Conference on Data Mining, pp. 51–60, Dallas, TX, USA, December 2013.
View at: Publisher Site | Google Scholar
W. Cai, Y. Zhang, S. Zhou, W. Wang, C. Ding, and X. Gu, “Active learning for support vector machines with maximum model change,” Machine Learning and Knowledge Discovery in Databases, vol. 8724, pp. 211–226, 2014.
View at: Publisher Site | Google Scholar
W. Cai, Y. Zhang, Y. Zhang et al., “Active learning for classification with maximum model change,” ACM Transactions on Information Systems, vol. 36, no. 2, pp. 1–28, 2018.
View at: Publisher Site | Google Scholar
W. Cai, M. Zhang, and Y. Zhang, “Batch mode active learning for regression with expected model change,” IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 7, pp. 1668–1681, 2017.
View at: Publisher Site | Google Scholar
A. Freytag, E. Rodner, and J. Denzler, “Selecting influential examples: active learning with expected model output changes,” in Proceedings of the European Conference on Computer Vision, Computer Vision – ECCV 2014, pp. 562–577, Springer, Zurich, Switzerland, September 2014.
View at: Google Scholar
J. O’Neill, S. Jane Delany, and B. MacNamee, “Model-free and model-based active learning for regression,” Advances in Intelligent Systems and Computing, Springer, Cham, pp. 375–386, 2017.
View at: Publisher Site | Google Scholar
S. H. Park and S. B. Kim, “Robust expected model change for active learning in regression,” Applied Intelligence, vol. 50, no. 2, pp. 296–313, 2020.
View at: Publisher Site | Google Scholar
H. Yu and S. Kim, “Passive sampling for regression,” in Proceedings of the IEEE International Conference on Data Mining, pp. 1151–1156, Sydney, Australia, December 2010.
View at: Publisher Site | Google Scholar
L. F. O. Chamon and A. Ribeiro, “Greedy sampling of graph signals,” IEEE Transactions on Signal Processing, vol. 66, no. 1, pp. 34–47, 2018.
View at: Publisher Site | Google Scholar
D. R. Wu, C. T. Lin, and J. Huang, “Active learning for regression using greedy sampling,” Information Sciences, vol. 474, pp. 90–105, 2019.
View at: Publisher Site | Google Scholar
D. Wu, “Pool-based sequential active learning for regression,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 5, pp. 1348–1359, 2019.
View at: Publisher Site | Google Scholar
Z. Liu and D. Wu, “Unsupervised pool-based active learning for linear regression,” 2020, http://export.arxiv.org/pdf/2001.05028.
View at: Google Scholar
D. Ginsbourger, R. Le Riche, and L. Carraro, “Kriging is well-suited to parallelize optimization,” Computational Intelligence in Expensive Optimization Problems, Springer, Berlin, Heidelberg, GER, pp. 131–162, 2010.
View at: Publisher Site | Google Scholar
A. Chaudhuri and R. T. Haftka, “Efficient global optimization with adaptive target setting,” AIAA Journal, vol. 52, no. 7, pp. 1573–1578, 2014.
View at: Publisher Site | Google Scholar
J. Mockus, V. Tiesis, and A. Zilinskas, “The application of Bayesian methods for seeking the extremum,” Towards Global Optimization, vol. 2, pp. 117–129, 1978.
View at: Google Scholar
W. B. Powell and I. O. Ryzhov, “Optimal learning,” Wiley Series in Probability and Statistics, John Wiley & Sons, Hoboken, USA, 2012.
View at: Publisher Site | Google Scholar
A. E. Hoerl and R. W. Kennard, “Ridge regression: applications to nonorthogonal problems,” Technometrics, vol. 12, no. 1, pp. 69–82, 1970.
View at: Publisher Site | Google Scholar
R. Tibshirani, “Regression shrinkage and selection via the lasso: a retrospective,” Journal of the Royal Statistical Society: Series B, vol. 73, no. 3, pp. 273–282, 2011.
View at: Publisher Site | Google Scholar
H. Zou and T. Hastie, “Regularization and variable selection via the elastic net,” Journal of the Royal Statistical Society: Series B, vol. 67, no. 2, pp. 301–320, 2005.
View at: Publisher Site | Google Scholar
R. L. Iman and J. M. Davenport, “Approximations of the critical region of the fbietkan statistic,” Communications in Statistics - Theory and Methods, vol. 9, no. 6, pp. 571–595, 1980.
View at: Publisher Site | Google Scholar
J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006.
View at: Google Scholar

Copyright

Copyright © 2022 Tianxin Zong et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

320

Downloads

385

Citations

Journal of Electrical and Computer Engineering

Big Data Analytics in Mobile Information Systems for Advanced Computing

[Retracted] Active Learning Query Strategies for Linear Regression Based on Efficient Global Optimization

Abstract

1. Introduction

2. Related Work

2.1. Existing Pool-Based ALR Query Strategies

2.1.1. Supervised ALR Query Strategies

2.1.2. Unsupervised ALR Query Strategies

2.2. Efficient Global Optimization

3. ALR Query Strategies Integrated with EGO

3.1. Combination of Supervised ALR and EGO

3.2. Combination of Unsupervised ALR and EGO

4. Results

4.1. Data Sources

4.2. Comparison Algorithm

4.3. Evaluation Process

4.4. Experimental Result on Ridge

4.5. Experimental Results on Lasso and Enet

4.6. Statistical Analysis

4.7. Parameter c Sensitivity

4.8. Visualization of Sample Selection

5. Conclusions

Data Availability

Conflicts of Interest

Acknowledgments

Supplementary Materials

References

Copyright