Reinforcement Learning-Based Genetic Algorithm in Optimizing Multidimensional Data Discretization Scheme

Chen, Qiong; Huang, Mengxing; Xu, Qiannan; Wang, Hao; Wang, Jinghui

doi:https://doi.org/10.1155/2020/1698323

Mathematical Problems in Engineering

On this page

Abstract Introduction Background and Related Work Results and Discussion Conclusion Data Availability Disclosure Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2020 | Article ID 1698323 | https://doi.org/10.1155/2020/1698323

Reinforcement Learning-Based Genetic Algorithm in Optimizing Multidimensional Data Discretization Scheme

Qiong Chen,^1,2Mengxing Huang ,^1,2Qiannan Xu,³Hao Wang,⁴and Jinghui Wang^1,2

Academic Editor: Bogdan Dumitrescu

Received22 Nov 2019

Revised19 Feb 2020

Accepted28 Feb 2020

Published30 Mar 2020

Abstract

Feature discretization can reduce the complexity of data and improve the efficiency of data mining and machine learning. However, in the process of multidimensional data discretization, limited by the complex correlation among features and the performance bottleneck of traditional discretization criteria, the schemes obtained by most algorithms are not optimal in specific application scenarios and can even fail to meet the accuracy requirements of the system. Although some swarm intelligence algorithms can achieve better results, it is difficult to formulate appropriate strategies without prior knowledge, which will make the search in multidimensional space inefficient, consume many computing resources, and easily fall into local optima. To solve these problems, this paper proposes a genetic algorithm based on reinforcement learning to optimize the discretization scheme of multidimensional data. We use rough sets to construct the individual fitness function, and we design the control function to dynamically adjust population diversity. In addition, we introduce a reinforcement learning mechanism to crossover and mutation to determine the crossover fragments and mutation points of the discretization scheme to be optimized. We conduct simulation experiments on Landsat 8 and Gaofen-2 images, and we compare our method to the traditional genetic algorithm and state-of-the-art discretization methods. Experimental results show that the proposed optimization method can further reduce the number of intervals and simplify the multidimensional dataset without decreasing the data consistency and classification accuracy of discretization.

1. Introduction

The rapid development of the internet of things has produced massive amounts of large-scale data [1–5]. These are mainly from various types of sensors, with high-dimensional, incomplete, random, fuzzy, and strong interference and other characteristics [6]. Despite the growing body of artificial intelligence research, how to extract and analyze valuable information from these massive amounts of complex sensor data is still a huge challenge in the field of artificial intelligence [7–9]. As one of the most influential data preprocessing technologies, feature discretization can reduce the complexity of data by transforming the continuous features in massive data to discrete features and obtain shorter, more accurate, and more comprehensible rules, so as to improve the efficiency of data mining and machine learning [10–16]. Therefore, feature discretization plays a key role in the application of artificial intelligence technology to these data.

With the development of artificial intelligence, more and more scholars are studying feature discretization [17–19]. Obtaining the optimal discretization scheme has been proved to be an NP complete problem [20]. The choice of a discretization method for a dataset will restrict the performance and accuracy of a posterior learning task. Discretization technology can be classified as either supervised or unsupervised, according to whether the data contain category information [21].

EqualWidth and EqualFrequency are two commonly used unsupervised discretization methods [22]. They divide the entire attribute according to a given interval length and frequency. Although they are simple and convenient, they lead to uneven data distribution and loss of some important information. Supervised discretization has the advantage of making full use of the class label and target attribute information, and it facilitates finding the appropriate breakpoint position compared to unsupervised discretization.

1R is the simplest supervised discretization algorithm [23]. It uses a greedy strategy to divide the attribute value range into intervals, each corresponding only to a decision class. However, because the partition standard of the interval is too simple, lacks flexibility, and does not consider the correlation between features, it cannot guarantee that the compatibility of an information system after discretization will not be destroyed.

The information entropy-based method is based on the minimum length description principle (MDLP), and it uses the measure of information gain to determine the breakpoint [24]. Although it can largely ensure the consistency of samples in the interval, it is difficult to filter the noise by setting the threshold value of the partition.

ChiMerge [25], Chi2 [26], and Extended Chi2 [27] discriminate and merge adjacent intervals by measuring the degree of association between condition attributes and target attributes. They have the advantage that the structures of adjacent intervals are distinct, but they are sensitive to parameters.

CADD [28], CAIM [29], and CACC [30] use statistics to quantify class attribute correlation, which they can maximize after discretization, and they can obtain the minimum number of discrete intervals as much as possible. However, they only achieve the best discretization for a single attribute interval, lack the description of all of the data, do not consider the consistency of the data either before or after discretization, and will inevitably lose important information from the original data.

These mainstream methods are obviously based on specific division criteria to achieve the discretization of continuous features. However, in the process of multidimensional data discretization, the distribution of target attribute values is usually difficult to know, and the features have complex correlations. In addition, the relatively fixed division criteria cannot provide a comprehensive measure of discrete intervals, and there will be some defects. Therefore, the discretization schemes obtained by most of the algorithms are not optimal in specific application scenarios, or they may even fail to meet the accuracy requirements of the system. A PSO algorithm was applied to feature selection based on discretization, which can generate more powerful and compact representations in high-dimensional datasets, and thus achieve better classification performance [31]. ACO [32] is used to solve the problem of discretization of continuous features to obtain more concise decision rules and higher prediction accuracy. RS-GA is a mature discretization method, which uses the individual fitness function based on rough sets to evaluate the uncertainty of an information system in a genetic algorithm and searches for the optimal discretization scheme through individual evolution [33]. Although these swarm intelligence algorithms can achieve better results, it is difficult to formulate appropriate strategies without prior knowledge, which will make the search in multidimensional space inefficient, consume computing resources, and easily fall into local optima.

Aiming at these problems, this paper proposes a reinforcement learning-based genetic algorithm (RLGA) to optimize the discretization scheme of multidimensional data. First, we binary code the attribute values of the multidimensional data and initialize the population. The binary code method can build an efficient mathematical model suitable for the problem of feature discretization. Second, it is difficult to form a proper strategy without prior knowledge as guidance, which causes the search space to easily fall into local optima. We use rough sets [34] to construct individual fitness functions and a design control function to dynamically adjust the diversity of the population. Then, we introduce a reinforcement learning mechanism [35] to crossover and mutation to determine the crossover fragments and mutation points of the discretization scheme to be optimized. We compare our method to state-of-the-art discretization methods on GF-2 and Landsat 8 images. Experimental results show that the proposed method can reduce the number of intervals and simplify the multidimensional dataset without decreasing the data consistency and classification accuracy of a discretization scheme.

The remainder of this paper is arranged as follows. Section 2 describes basic concepts and reviews some related work. Section 3 explains the algorithm flow of the proposed work. Section 4 introduces the experimental environment and datasets. We analyze and discuss the experimental results in Section 5. Section 6 summarizes this paper and discusses future research.

We introduce the concept of feature discretization and present simple descriptions of rough sets, genetic algorithms, and reinforcement learning mechanisms. We also analyze problems that occur in the discretization of multidimensional data in the optimization process.

2.1. Definitions of Feature Discretization

Feature discretization is the process of dividing a continuous attribute value (also called a continuous feature value) into a finite number of intervals according to a rule, and associating these intervals with a set of discrete values [10]. Considering the problem of -dimensional data classification, the discretization algorithm divides the attribute values on the -th dimension into discrete, nonintersecting intervals:where and are the minimum and maximum attribute values. All of the values are arranged in ascending order in , which is called a discretization scheme on the th dimension. represents the whole discretization scheme of -dimensional data. Obviously, the search space of -dimensional data feature discretization is formed by all of the candidate breakpoints of each dimension, which are different attribute values of each dimension in the training set.

2.2. Genetic Algorithm Using Binary Coding

As a global optimization probability evolution algorithm, a genetic algorithm [36] has inherent implicit parallelism and a strong global search ability. It achieves good performance on many optimization problems [37]. We use binary encoding to encode the candidate breakpoint; the values 1 and 0 represent that the breakpoint is selected or discarded, respectively. Assuming that is a candidate breakpoint set of -dimensional data in the th dimension, the chromosome structure in the genetic algorithm is shown in Figure 1, where colors represent different features in the -dimensional data. The length of each chromosome is . The set of selected candidate breakpoints is a discretization scheme, according to which attribute values are classified into discrete intervals formed by the candidate breakpoints in the set.

2.3. Fitness Function Based on Rough Set

A rough set [38] is based on the classification mechanism. It interprets the classification as indiscernible relations in the space of features, and these relations form the division of the space. Given decision table , where is a finite set of objects, i.e., a domain, is an attribute set including condition attribute set and decision attribute set . For each attribute subset , the indiscernible binary relation and the equivalent classes of attribute subset in domain are defined as

According to the abovementioned decision table , for each subset and equivalent class of attribute subset in , the lower and upper approximation sets of are defined as

Since the principle of selecting the optimal breakpoint set is to minimize the number of breakpoints without changing the indiscernible relations of the decision table, the fitness function should be determined by the number of breakpoints and the indiscernible relations:where is the number of breakpoints in the initial breakpoint set, is the number of breakpoints obtained after chromosome decoding, and is the change of the number of breakpoints. takes the value 0 or 1, which, respectively, mean that the indiscernibility of the decision table changes or does not change after discretization. The judgment process is shown in Algorithm 1.

	Input: Discretization scheme, original decision table
	Output: Indiscernible relationship change
	*initialize*: R_D = 1;
	begin
	for each category i do
	Compute the lower approximation set C₋(d_i) of the original decision table before discretization using equation (3);
	Compute the upper approximation set C⁻(d_i) of the original decision table before discretization using equation (4);
	end
	Discrete the original decision table by the discretization scheme;
	for each category i do
	Compute the lower approximation set C₋(d_i)′ of the original decision table after discretization using equation (3);
	Compute the upper approximation set C⁻(d_i)′ of the original decision table after discretization using equation (4);
	end
	for each category i do
	if C₋(d_i)′ ≠ C₋(d_i) or C⁻(d_i)′ ≠ C⁻(d_i) do
	R_D = 0;
	break;
	end
	end
	Return R_D;
	end

2.4. Reinforcement Learning

Reinforcement learning is a goal-driven, highly adaptive machine learning technique in the field of artificial intelligence [39], in which there are two basic elements: state and action. Performing an action in a certain state is a strategy. The learner must constantly explore to generate an optimal strategy. Different from supervised and unsupervised learning, it regards learning as a process of interaction between agents and the environment through exploration and evaluation. The operation mechanism is shown in Figure 2. The agent selects an action to be applied to the environment by sensing the current state of the environment. After the environment accepts the action, the state changes, and a reward is given to the agent. According to the new state of the environment, the agent continues to select the next action, and this is repeated until it reaches the terminated state. The goal of reinforcement learning is to maximize the accumulated rewards by adjusting strategies.

Q-learning [40] is one of the most representative model-free reinforcement learning techniques. In the current state, the agent selects the next action according to the corresponding value of each action using the strategy and updates at each step in the learning process aswhere and are, respectively, the current state and behavior, , and are the next state and behavior, is the action set, and . Q-learning has broad application prospects in solving complex control and decision-making problems [41, 42]. We use Q-learning to determine the cross fragments and mutation points of the discretization scheme to be optimized.

2.5. Main Challenges

The complexity of multidimensional data feature discretization increases sharply with the length of the attribute value interval and the association between attributes [36]. When using the genetic algorithm [43] to optimize the discretization scheme of multidimensional data, the main challenges are as follows.(1)Improper control of population diversity causes premature convergence.(2)Because multidimensional data contain many features, cross fragments and mutation points tend to focus on features with larger value intervals, which decrease the opportunity for breakpoints on other features to evolve.(3)Due to the complex correlations between features, the crossover and mutation operations are relatively blind without prior knowledge as guidance, making it highly likely that some high-quality fragments on features are destroyed in the next generation of evolutionary operations.

3. Reinforcement Learning-Based Genetic Algorithm

A genetic algorithm is a general method to search for the optimal solution in the field of artificial intelligence. We add a control function to the fitness function based on rough sets to dynamically adjust the diversity of the population. In addition, according to the characteristics of multidimensional data, we introduce a reinforcement learning mechanism to the crossover and mutation operation to determine the cross fragments and mutation points of the discretization scheme to be optimized, which greatly improves the accuracy and speed of convergence. Below, we discuss the flow of the algorithm.

3.1. Evolution of Discretization Scheme to Be Optimized

We perform evolutionary operations on the optimized discretization scheme and initial population, as shown in Figure 3. The global variable preserves the best individual of the population, while the local variable preserves the historical best individual of the discretization scheme to be optimized. In each iteration, the fitness of the current population is calculated, the optimal individual is obtained, and the global variable is updated. Then, the discretization scheme to be optimized and the optimal individual of the population carry out the cross and mutation operations based on reinforcement learning, and the local variable is updated. If the termination condition is not reached, then the population will continue to carry out ordinary evolutionary operations. Otherwise, by comparing the global variable and local variable, the individual with the largest fitness will be output.

3.2. Selection Operator Based on the Control Function

The selection operation is based on the evaluation of individual fitness in the population. Individuals with higher fitness are generally more likely to be selected. Roulette [44] is a simple, efficient, probability-based method that is often used to select individuals. If the population size is and the fitness value of individual is , then the probability that individual is selected is

However, individuals with large fitness are more likely to be selected, which may result in the destruction of population diversity. Since takes the value 0 or 1, individuals with fitness value 0 cannot be selected in the evolution of the population. Although these individuals are not feasible solutions at the initial stage, they may be close to the optimal feasible solution, and may eventually evolve into it. However, considering the randomness and evenness of the initial population, feasible solutions with = 1 and potential feasible solutions with = 0 tend to be quantitatively equivalent. Even with a small population, because of the complex correlation among the features of multidimensional data, there are far more potential feasible solutions with = 0 than with = 1. Therefore, when roulette is used for individual selection, the potential feasible solutions that are the majority of the population will be eliminated, thus destroying the diversity of the population. According to the abovementioned analysis, to ensure the diversity of the population in the early stage and accelerate convergence of individuals to the optimal solution in the late stage of evolution, we expand the fitness function by adding the control function on the original basis, and we determine the control factors according to the proportion of feasible solutions in the population in the current stage of evolution. The fitness function of chromosome is thus expanded to

The control function consists of control item and control factor , where is the proportion of the population consisting of potential feasible solutions with = 0, and is a function with as an independent variable. The expressions for and arewhere is the number of breakpoints in the initial breakpoint set, is the number of breakpoints obtained after chromosome decoding, is the change of the indiscernible relationship of the decision table after discretization by chromosome , is the number of classes, is the th class, and are, respectively, the lower and upper approximation sets of before discretization, and are, respectively, the lower and upper approximation sets of after discretization, and is a user-defined set comparison function. When and are equal, the function value is 1, and otherwise it is 0. Since each class corresponds to a lower and upper approximation set, the value range of is . The following points are made regarding the extended fitness function.(1)We want to prevent the unreasonable situation that when the number of breakpoints of the feasible solution with is equal to that of the potential feasible solution with , the fitness value of the former is less than or equal to that of the latter. Hence we require that index of the control item satisfies . In this paper, .(2)It can be seen that the value range of is actually discrete points from 0 to 1, with the interval . When , the indiscernibility of the decision table is destroyed after discretization. If , then = 0, and the significance of adding a control function for potential feasible solutions with is lost, since the control function has a smaller value than when takes the minimum nonnegative value . Consequently, we change the expression to . So, when , the value of the control function is not 0, and it is smaller than its value with . However, because the chromosome is binary coded, taking “2” as the base number and as the true number can ensure that the difference between the two control function values corresponding to and is not too large, and this controls the selection of individuals relatively reasonably.(3)When , and , indicating that this individual is a feasible solution, and the value of the control item is 0. When , and is an increasing function with as the independent variable. Consider that in the early stage of evolution, there are few or no feasible solutions in the population, i.e., is large. At this time, should be relatively large, so the potential feasible solution can be selected with a large probability in roulette, and the search is gradually guided to the area of feasible solutions. As evolution progresses, more and more feasible solutions appear in the population, i.e., becomes smaller and smaller. Accordingly, should become smaller and smaller, accelerating the movement from feasible to optimal solutions.

3.3. Crossover Operator Based on Q-Learning

The crossover operation in a genetic algorithm is the exchange of some genes between two matched chromosomes in a certain way, so as to form two new individuals. However, multidimensional data contain many features, and cross fragments tend to focus on features with large value intervals; hence, the breakpoints on other features lose the chance to cross. However, there are complex correlations among features. Without prior knowledge as a guide, the cross operation of the discretization scheme to be optimized becomes blind, which causes some high-quality fragments to have a high probability of being destroyed. To this end, we use the decision-making ability of Q-learning to select some features of the discretization scheme to be optimized in each iteration for the cross operation.(1)State: according to the abovementioned analysis, the crossover operation will give each feature a certain probability of change. We define the set of changing features in the crossover operation as a state. Assuming that the multidimensional data have features, the search space is divided into states, each a combination of several features. For example, when , there are seven states: , , , , , , and , where is the th feature of multidimensional data, , and the elements in represent the features corresponding to the loci of the most recent crossover or mutation operation.(2)Action: since multidimensional data generally contain a large number of features, according to the abovementioned definition of states, the number of states will increase exponentially. The transition between two states corresponds to one action; so, many actions must be defined, which increases the computational complexity. According to the previous analysis, we mainly want to avoid the situation that in each crossover operation, cross segments focus on features with a larger value range, which makes some breakpoints lose the chance of crossing. In addition, some high-quality segments will be destroyed without prior knowledge as guidance. Therefore, for the current state, the next state to jump to after performing an action should be mainly considered from three aspects. First, the feature set of the next state is a subset of that of the previous state. Second, the feature set of the next state is a complement of that the previous state. Third, the intersection of the feature set of the next state and that of the previous state is not empty. Accordingly, there are three kinds of actions, represented by , , and . The algorithm jumps to a new state by performing actions on the current state, and executes cross operations on all of the features contained in the new state. Suppose is the current state and is the next state. represents a random jump to one of all of the subsets of the current state, is a random jump to one of all of the subsets of the complementary set of the current state, and is a random jump to a set whose intersection with the current state is not empty and is not a subset of the current state, as shown below. It is easy to see that the range of values for , , and covers all of the states.(3)Reward: to make a correct decision when selecting a feature set in each crossover operation so as to more quickly approach the optimal solution, we set a reward value for each state-action combination. The reward value is based on the change of individual fitness, which is mainly used to evaluate the search for the optimal solution of the algorithm. In the formula mentioned below, is the probability of jumping to the next state after performing action in the current state . This is related to the number of all of the possible states to jump to, which is . and are, respectively, the individual fitness of the current and next state, and is the historical best fitness of the discretization scheme to be optimized.

According to the designed reward value, we can update when the discretization scheme to be optimized has a cross operation.

3.4. Mutation Operator Based on Q-Learning

Like the crossover operator, we use the decision-making ability of Q-learning to select some features of the discretization scheme to be optimized in each iteration for the mutation operation. The state, action, and reward of the mutation and cross operation are consistent, the only difference being to change the cross operation on the feature to a mutation operation. They each maintain a Q-table. Figure 4 shows the update process of two Q-tables of a discretization scheme with three features in one iteration.

Both Q-tables are initialized to 0. After N iterations, the Q-tables corresponding to the cross and mutation operations are shown in Figures 4(a) and 4(b), respectively. Assuming that the current state is , action is selected for the cross operation. Accordingly, the state jumps to , and Q’s value of 1 is updated to 6 in cross operation. Then, state selects action for the mutation operation. Accordingly, the state randomly jumps to , and Q′ s value of 2 is updated to 3 in the mutation operation. In this way, the two Q-tables are updated in one iteration.

3.5. Flow of RLGA Algorithm

Algorithm 2 shows the flow of the proposed method. First, binary genetic coding is applied to the attribute values of the multidimensional data, and the state is generated according to the number of features. Then, in the current state of the discretization scheme to be optimized, the greedy algorithm is used to select the action, jump to the next state, and cross operate with the global optimal individual on the corresponding features. At the same time, the reward value is evaluated according to the fitness value after the cross operation, whose Q-table is then updated. Similarly, in the current state, an action is selected to continue the mutation operation, whose Q-table is updated. The population performs the conventional genetic operation and saves the global optimal individual in each iteration. Finally, the program outputs the maximum value of both the global and local variable. While the algorithm optimizes the given discretization scheme, other individuals of the population evolve to enlarge the search scope and improve the probability of obtaining the optimal solution.

	Input: Multidimensional data discretization scheme
	Output: Optimal discretization scheme
	Initialize: global variable = 0, local variable = 0, crossover Q-Table = null, mutation Q-Table = null, t = 0;
	begin
	Get the initial breakpoints of the multidimensional data by sorting the values of each feature and removing duplicate values;
	Binary encode the initial breakpoints of multidimensional data according to the method in Part B of Section2;
	Randomly generate initial population P(t);
	Calculate the fitness of each individual in P(t) using equation (8);
	Update global variable with the optimal individual fitness value in P(t);
	Generate state set based on the number of features of multidimensional data according to the definition of state in Part C of Section3;
	Choose a state from the state set as the initial state S(t);
	while t is less than the user’s termination iterations do
	Choose an action from the action set {G, H, I} by e-greedy strategy according to the definition of action in Part C of Section3;
	Execute the selected action on the current state S(t) to jump to the next state S(t + 1);
	Perform crossover operation with global variable on the features contained in state S(t + 1);
	Calculate the fitness of the multidimensional data discretization scheme after crossover operation using equation (5);
	Measure the corresponding reward using equation (11) according to the definition of reward in Part C of Section3;
	Update crossover Q-Table using equation (6);
	if the fitness of the multidimensional data discretization scheme > local variable do
	Update local variable with the fitness of the multidimensional data discretization scheme;
	end
	Perform crossover operation in P(t);
	Calculate the fitness of each individual in P(t) using equation (8);
	Update global variable with the optimal individual fitness value in P(t);
	S(t) = S(t + 1);
	Choose an action from the action set {G, H, I} by e-greedy strategy according to the definition of action in Part C of Section3;
	Execute the selected action on the current state S(t) to jump to the next state S(t + 1);
	Perform mutation operation on the features contained in state S(t + 1)
	Calculate the fitness of the multidimensional data discretization scheme after mutation operation using equation (5);
	Measure the corresponding reward using equation (11) according to the definition of reward in Part C of Section3;
	Update mutation Q-Table using equation (6);
	if the fitness of the multidimensional data discretization scheme > local variable do
	Update local variable with the fitness of the multidimensional data discretization scheme;
	end
	Perform mutation operation in P(t);
	Calculate the fitness of each individual in P(t) using equation (8);
	Update global variable with the optimal individual fitness value in P(t);
	t = t + 1;
	end
	Return Max(global variable, local variable);
	end

4. Experimental Design

We introduce the experimental data source, experimental environment configuration, and dataset used in the experiment.

4.1. Data Source

The experimental data are from a Landsat 8 satellite image in the coastal area of the South China Sea on February 22, 2018, as shown in Figure 5. The image consists of seven bands. In the experiment, the objects on the image are divided into five categories: impervious surface, construction, bare land, water, and vegetation.

4.2. Configuration of Experimental Environment

To verify the effectiveness of the algorithm in this paper, comparative experiments are carried out using an Intel Core i5-5200U [email protected], 12 GB memory, and 512 GB hard disk. The visualization, programming, simulation, testing, and calculation are realized in MATLAB R2016a. The radiometric calibration and atmospheric correction of images, training of classifiers for discrete results, and comparison of classification prediction accuracy are completed in an ENVI5.3 environment.

4.3. Preparation of Experimental Datasets

We randomly select several areas from the image, covering the five categories specified above and integrate them into a set of experimental samples containing training and test sets. The training set includes 6331 samples, consisting of 935 impervious surface samples, 936 construction samples, 958 bare land samples, 2324 water samples, and 1178 vegetation samples. We sort the pixel values of the training set and delete duplicate values in each band to obtain the initial breakpoints of the seven bands, which are 1403, 1429, 1680, 1869, 2402, 2530, and 2240, for a total of 13553 breakpoints. We discretize this initial set and carry out the comparative experiments.

We compare the proposed method to the classical genetic algorithm (GA) [33] based on the consistency principle of the decision system. Then, we compare the optimal set of breakpoints obtained by our method with those of current mainstream supervised discretization methods, such as EDiRa [45], ChiMerge [46], 1R [23], NCAIC [47], FUDC [48], Cramer’s V-Test [49], and Chi2 [50], mainly on the evaluation of the number of intervals and data consistency.

In addition, we use the proposed method to optimize the discretization results of the MFD-mvtR algorithm [51]. Finally, we train the neural network classifiers with the discretized samples of all of the methods, and verify the effectiveness of our method by comparing the classification accuracy of each method.

5. Results and Discussion

We compare the performance of our method with that of the classical genetic algorithm based on the consistency principle of the decision system, and we evaluate the results of seven state-of-the-art discretization methods on the Landsat 8 image. We also use our method to optimize the discretization results of MFD-mvtR [51] on the Gaofen-2 image. Finally, the effectiveness of our method is verified by comparing the classification accuracy of each method.

5.1. RLGA versus GA

We set the population size of the two algorithms at 30 and the number of iterations at 500, and we run them 10 times independently. Figure 6 shows the number of iterations of the two algorithms to reach the theoretical optimal solution in 10 independent experiments. Table 1 compares the convergence rates of the two algorithms. The search efficiency is expressed aswhere is the average number of iterations to obtain the optimal solution, is the total number of iterations, and is the search efficiency, i.e., the convergence speed. The larger the value, the faster the convergence.

It can be seen from Table 1 that the average number of iterations for GA to obtain the optimal solution is 425.3, and the search efficiency is 0.149. These numbers correspond, respectively, to 338.6 and 0.323 for our method, an obvious improvement in performance.

5.2. RLGA versus Mainstream Discretization Algorithms

We compare the optimal set of breakpoints obtained by our method to mainstream supervised discretization methods, including EDiRa [45], ChiMerge [46], 1R [23], NCAIC [47], FUDC [48], Cramer’s V-Test [49], and Chi2 [50], on the number of intervals and data consistency.

Figure 7 shows the number of discrete intervals obtained by the eight algorithms in each band. RLGA obtains the minimum number of breakpoints in each band. Table 2 compares the overall number of intervals and data consistency of the eight algorithms. We can see that the number of discrete intervals obtained by RLGA is 2247, which is the least among all of the algorithms, and there is no data error. EDiRa obtains 3909 discrete intervals with 4 data errors. The number of discrete intervals obtained by ChiMerge is 4947, and the number of data errors is also 4. The number of discrete intervals obtained by 1R is 3053, which is the least excepted for RLGA, but it has the most data errors among all of the algorithms, at 31. The number of discrete intervals obtained by NCAIC is the largest among all of the algorithms, at 5041, with 4 data errors. FUDC obtains 4072 discrete intervals, with 4 data errors. Cramer’s V-Test and Chi2 both obtain relatively small numbers of discrete intervals, 3858 and 3538, respectively, but there are also more data errors, 7 and 19, respectively. Considering the number of discrete intervals and the number of data errors, RLGA has the best discretization quality.

5.3. Optimization of Breakpoints on Gaofen-2 Image

We use RLGA to optimize the discretization results obtained by the MFD-mvtR algorithm on the Gaofen-2 image [51]. Considering that this image contains only four bands, the search space is divided into 15 states, which is less than the 127 states obtained from 7 bands of the Landsat 8 image. Therefore, we can set the number of iterations of RLGA to only 100 to allow the algorithm to fully learn. The number of optimized breakpoints is shown in Tables 3 and 4.

The MFD-mvtR algorithm [51] obtains 1345 breakpoints with 5 data errors on the Gaofen-2 image. Compared to other mainstream algorithms, the results of discretization are satisfactory. We take the results of MFD-mvtR as the discretization scheme to be optimized, use RLGA to perform the cross operation based on reinforcement learning with the optimal individuals in the population, and carry out its own mutation operation based on reinforcement learning. By further optimizing the results of MFD-mvtR, we can see that the numbers of breakpoints in the four bands are reduced to 310, 301, 222, and 198, respectively. The total number of breakpoints is 314 less than before optimization, and the number of data errors is reduced to 0.

Table 5 shows the classification accuracy of the neural network after training the discrete feature sets obtained by the abovementioned algorithms. We can see that after optimizing the discretization scheme of MFD-mvtR, the classification accuracy obtained by confusion matrix is about 6 percentage points higher than the original, and the kappa coefficient is 0.8925. Our method reduces the number of breakpoints in the discretization scheme of MFD-mvtR to a certain extent, while ensuring that there is no data error. It can simplify the dataset and effectively identify the five types of areas of impervious surface, construction, bare land, water, and vegetation on the image after training the classifier by the optimized discrete feature set, which improves the performance of the classifier.

6. Conclusion and Future Work

In the process of multidimensional data discretization, due to the complex correlation among features and the performance bottleneck of the traditional discretization criteria, most discretization schemes obtained by the algorithms are not optimal in specific application scenarios or they may even fail to meet the accuracy requirements of the system. Some swarm intelligence algorithms can achieve better results, but without prior knowledge as a guide, it is difficult to formulate appropriate strategies, which will cause an inefficient search in multidimensional space, consume many computing resources, and easily fall into local optimization. To solve these problems, this paper proposes a reinforcement learning-based genetic algorithm to optimize the discretization of multidimensional data. First, we binary code the attribute values of the multidimensional data and initialize the population. Second, we use rough sets to construct individual fitness functions and design control functions to dynamically adjust the diversity of the population. Then, we introduce the Q-learning reinforcement learning mechanism to the crossover and mutation operations to determine the crossover fragments and mutation points of the discretization to be optimized. We conduct simulation experiments on Landsat 8 and Gaofen-2 images to compare RLGA to the traditional genetic algorithm and state-of-the-art discretization methods. The experimental results show that our method can reduce the number of breakpoints and simplify the multidimensional dataset without decreasing the data consistency and classification accuracy of a discretization scheme.

Future research work includes the following: (1) test and improve the proposed method on different multidimensional datasets, expand its application scope, and make it more practical; (2) to ease the problems of high-dimensional datasets, improve the algorithm using deep Q-learning technology; and (3) improve the performance of the deep neural network by using RLGA to optimize its parameters.

Data Availability

The Landsat 8 and Gaofen-2 processing data used to support the findings of this study are included within the article.

Disclosure

The sponsors had no role in the design, execution, interpretation, or writing of the study.

Conflicts of Interest

The authors declare no conflicts of interest.

Acknowledgments

The authors would like to thank the support of the laboratory, university, and government. This work was supported by Hainan Provincial Natural Science Foundation of China (Grant no: 2019CXTD400) and the National Key Research and Development Program of China (Grant no: 2018YFB140400).

References

X. Zeng, G. Xu, X. Zheng, Y. Xiang, and W. Zhou, “E-AUA: an efficient anonymous user authentication protocol for mobile IoT,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 1506–1519, 2019.
View at: Publisher Site | Google Scholar
L. Li, G. Xu, L. Jiao et al., “A secure random key distribution scheme against node replication attacks in industrial wireless sensor systems,” IEEE Transactions on Industrial Informatics (Early Access), vol. 16, no. 3, pp. 2091–2101, 2020.
View at: Publisher Site | Google Scholar
Y. Yin, L. Chen, Y. Xu, and J. Wan, “Location-aware service recommendation with enhanced probabilistic matrix factorization,” IEEE Access, vol. 6, pp. 62815–62825, 2018.
View at: Publisher Site | Google Scholar
H. Kang and C. Chen, “Fruit detection and segmentation for apple harvesting using visual sensor in orchards,” Sensors, vol. 19, no. 20, p. 4599, 2019.
View at: Publisher Site | Google Scholar
X. Yan, L. Wen, and L. Gao, “A fast and effective image preprocessing method for hot round steel surface,” Mathematical Problems in Engineering, vol. 2019, Article ID 9457826, pp. 1–14, 2019.
View at: Publisher Site | Google Scholar
N. Kafli and K. Isa, “Internet of Things (IoT) for measuring and monitoring sensors data of water surface platform,” in Proceedings of the 2017 IEEE 7th International Conference on Underwater System Technology: Theory and Applications (USYS), Kuala Lumpur, Malaysia, December 2017.
View at: Publisher Site | Google Scholar
G. Xu, W. Wang, L. Jiao et al., “SoProtector: safeguard privacy for native SO files in evolving mobile IoT applications,” IEEE Internet of Things Journal, p. 1, 2019.
View at: Publisher Site | Google Scholar
Y. Zhong, S. Fong, R. Hu, R. Wong, and W. Lin, “A novel sensor data pre-processing methodology for the internet of things using anomaly detection and transfer-by-subspace-similarity transformation,” Sensors, vol. 19, no. 20, p. 4536, 2019.
View at: Publisher Site | Google Scholar
H. S. Hassanein and M. A. O. Sharief, “big sensed data challenges in the Internet of Things,” in Proceedings of the 2017 13th International Conference on Distributed Computing in Sensor Systems (DCOSS), Ottawa, Canada, 2017.
View at: Publisher Site | Google Scholar
H. Liu, F. Hussain, C. L. Tan, and M. Dash, “Discretization: an enabling technique,” Data Mining and Knowledge Discovery, vol. 6, no. 4, pp. 393–423, 2002.
View at: Publisher Site | Google Scholar
S. Ramírez-Gallego, S. García, H. Mouriño-Talín et al., “Data discretization: taxonomy and big data challenge,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 6, no. 1, pp. 5–21, 2016.
View at: Publisher Site | Google Scholar
M. J. Abadi, L. Luceri, M. Hassan, C. T. Chou, and M. Nicoli, “A cooperative machine learning approach for pedestrian navigation in indoor IoT,” Sensors, vol. 19, no. 21, p. 4609, 2019.
View at: Publisher Site | Google Scholar
Y.-C. Lin, C.-L. Lin, M.-D. Tsai, and L.-S. Chou, “Discretization of object-based lidar features for land cover classification,” in Proceedings of the 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Fort Worth, TX, USA, July 2017.
View at: Publisher Site | Google Scholar
A. Gomaa, M. M. Abdelwahab, M. Abo-Zahhad, T. Minematsu, and R. I. Taniguchi, “Robust vehicle detection and counting algorithm employing a convolution neural network and optical flow,” Sensors, vol. 19, no. 20, p. 4588, 2019.
View at: Publisher Site | Google Scholar
S. C. Tan, “Improving association rule mining using clustering-based discretization of numerical data,” in Proceedings of the 2018 International Conference on Intelligent and Innovative Computing Applications (ICONIC), Plaine Magnien, Plaine Magnien, Mauritius, December 2018.
View at: Publisher Site | Google Scholar
D. S. D. Cunha, R. S. Xavier, D. G. Ferrari, F. G. Vilasbôas, and L. N. de Castro, “Bacterial colony algorithms for association rule mining in static and stream data,” Mathematical Problems in Engineering, vol. 2018, Article ID 4676258, 14 pages, 2018.
View at: Publisher Site | Google Scholar
R. Zamudio-Reyes, N. Cruz-Ramírez, and E. Mezura-Montes, “A multivariate discretization algorithm based on multiobjective optimization,” in Proceedings of the 2017 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA, December 2017.
View at: Publisher Site | Google Scholar
D. Maryono, P. Hatta, and R. Ariyuana, “Implementation of numerical attribute discretization for outlier detection on mixed attribute dataset,” in Proceedings of the 2018 International Conference on Information and Communications Technology (ICOIACT), Yogyakarta, Indonesia, March 2018.
View at: Publisher Site | Google Scholar
A. E. Rad, H. Nezamabadi-pour, and M. Eftekhari, “Improving LAIM discretization method for multi-label data using evolution strategy,” in Proceedings of the 2018 3rd Conference on Swarm Intelligence and Evolutionary Computation (CSIEC), Bam, Iran, March 2018.
View at: Publisher Site | Google Scholar
B. S. Chlebus and S. H. Nguyen, “On finding optimal discretizations for two attributes,” Rough Sets and Current Trends in Computing, Springer, Berlin, Germany, 1998.
View at: Publisher Site | Google Scholar
S. Garcia, J. A. Luengo, J. A. Sáez, V. López, and F. Herrera, “A survey of discretization techniques: taxonomy and empirical analysis in supervised learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 25, no. 4, pp. 734–750, 2013.
View at: Publisher Site | Google Scholar
A. K. C. Wong and D. K. Y. Chiu, “Synthesizing statistical knowledge from incomplete mixed-mode data,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-9, no. 6, pp. 796–805, 1987.
View at: Publisher Site | Google Scholar
C. H. Robert, “Very simple classification rules perform well on most commonly used datasets,” Machine Learning, vol. 11, no. 1, pp. 63–90, 1993.
View at: Google Scholar
K. Shehzad, “EDISC: a class-tailored discretization technique for rule-based classification,” IEEE Transactions on Knowledge and Data Engineering, vol. 24, no. 8, pp. 1435–1447, 2012.
View at: Publisher Site | Google Scholar
R. Kerber, “ChiMerge: discretization of numeric attributes,” in Proceedings of the AAAI’92 Proceedings of the tenth national conference on Artificial intelligence, San Jose, CA, USA, July 1992.
View at: Google Scholar
H. Liu and R. Setiono, “Feature selection via discretization,” IEEE Transactions on Knowledge and Data Engineering, vol. 9, no. 4, pp. 642–645, 1997.
View at: Publisher Site | Google Scholar
C.-T. Su and J.-H. Hsu, “An extended Chi2 algorithm for discretization of real value attributes,” IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 3, pp. 437–441, 2005.
View at: Publisher Site | Google Scholar
J. Y. Ching, A. K. C. Wong, and K. C. C. Chan, “Class-dependent discretization for inductive learning from continuous and mixed-mode data,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 7, pp. 641–651, 1995.
View at: Publisher Site | Google Scholar
L. A. Kurgan and K. J. Cios, “CAIM discretization algorithm,” IEEE Transactions on Knowledge and Data Engineering, vol. 16, no. 2, pp. 145–153, 2004.
View at: Publisher Site | Google Scholar
C.-J. Tsai, C.-I. Lee, and W.-P. Yang, “A discretization algorithm based on class-attribute contingency coefficient,” Information Sciences, vol. 178, no. 3, pp. 714–731, 2008.
View at: Publisher Site | Google Scholar
B. Tran, B. Xue, and M. Zhang, “A new representation in PSO for discretization-based feature selection,” IEEE Transactions on Cybernetics, vol. 48, no. 6, pp. 1733–1746, 2018.
View at: Publisher Site | Google Scholar
W. Zhu, J. Wang, Y. Zhang, and L. Jia, “A discretization algorithm based on information distance criterion and ant colony optimization algorithm for knowledge extracting on industrial database,” in Proceedings of the 2010 IEEE International Conference on Mechatronics and Automation, Xi’an, China, August 2010.
View at: Publisher Site | Google Scholar
C.-Y. Chen, Z.-G. Li, S.-Y. Qiao, and S.-P. Wen, “Study on discretization in rough set based on genetic algorithm,” in Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No. 03EX693), Xi’an, China, November 2003.
View at: Publisher Site | Google Scholar
Z. Pawlak, Rough Sets: Theoretical Aspects of Reasoning about Data, Kluwer Academic Publishers, Norwell, MA, USA, 1992.
V. Mnih, K. Kavukcuoglu, D. Silver et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
View at: Publisher Site | Google Scholar
S. Ramirez-Gallego, S. Garcia, J. M. Benítez, and F. Herrera, “Multivariate discretization based on evolutionary cut points selection for classification,” IEEE Transactions on Cybernetics, vol. 46, no. 3, pp. 595–608, 2016.
View at: Publisher Site | Google Scholar
D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley Professional, Boston, MA, USA, 1989.
Z. Pawlak, “Rough set theory and its applications to data analysis,” Cybernetics and Systems, vol. 29, no. 7, pp. 661–688, 1998.
View at: Publisher Site | Google Scholar
M. Kusy and R. Zajdel, “Application of reinforcement learning algorithms for the adaptive computation of the smoothing parameter for probabilistic neural network,” IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no. 9, pp. 2163–2175, 2015.
View at: Publisher Site | Google Scholar
C. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no. 3-4, pp. 279–292, 1992.
View at: Publisher Site | Google Scholar
V. Mnih, K. Kavukcuoglu, D. Silver et al., “Playing atari with deep reinforcement learning,” 2013, https://arxiv.org/abs/1312.5602.
View at: Google Scholar
L. Lv, S. Zhang, D. Ding, and Y. Wang, “Path planning via an improved DQN-based learning policy,” IEEE Access, vol. 7, pp. 67319–67330, 2019.
View at: Publisher Site | Google Scholar
K. Sriwanna, T. Boongoen, and N. Iam-On, “An evolutionary cut points search for graph clustering-based discretization,” in Proceedings of the 2016 13th International Joint Conference on Computer Science and Software Engineering (JCSSE), Khon Kaen, Thailand, July 2016.
View at: Publisher Site | Google Scholar
L. Zhang, H. Chang, and R. Xu, “Equal-width partitioning roulette wheel selection in genetic algorithm,” in Proceedings of the 2012 Conference on Technologies and Applications of Artificial Intelligence, Tainan, Taiwan, November 2012.
View at: Publisher Site | Google Scholar
C. R. de Sá, C. Soares, and A. Knobbe, “Entropy-based discretization methods for ranking data,” Information Sciences, vol. 329, no. C, pp. 921–936, 2016.
View at: Publisher Site | Google Scholar
Z. Ali and W. Shahzad, “Comparative study of discretization methods on the performance of associative classifiers,” in Procedings of the 2016 International Conference on Frontiers of Information Technology (FIT), Islamabad, Pakistan, December 2016.
View at: Google Scholar
D. Yan, D. Liu, and Y. Sang, “A new approach for discretizing continuous attributes in learning systems,” Neurocomputing, vol. 133, no. 10, pp. 507–511, 2014.
View at: Publisher Site | Google Scholar
G. Zhang, Z. Wu, and L. Yi, “A remote sensing feature discretization method accommodating uncertainty in classification systems,” in Proceedings of the 8th International Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Sciences, Shanghai, China, June 2008.
View at: Google Scholar
B. Wu, L. Zhang, and Y. Zhao, “Feature selection via cramer’s V-test discretization for remote-sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 52, no. 5, pp. 2593–2606, 2014.
View at: Publisher Site | Google Scholar
W. Qu, D. Yan, S. Yu, H. Liang, M. Kitsuregawa, and K. Li, “A novel Chi2 algorithm for discretization of continuous attributes,” in Proceedings of the APWeb’08 Proceedings of the 10th Asia-Pacific web conference on Progress in WWW research and development, Shenyang, China, April 2008.
View at: Google Scholar
M. Huang, Q. Chen, and H. Wang, “A multivariable optical remote sensing image feature discretization method applied to marine vessel targets recognition,” Multimedia Tools and Applications, vol. 79, no. 7-8, pp. 1–22, 2019.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2020 Qiong Chen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

7268

Downloads

1617

Citations

Mathematical Problems in Engineering

Reinforcement Learning-Based Genetic Algorithm in Optimizing Multidimensional Data Discretization Scheme

Abstract

1. Introduction

2. Background and Related Work

2.1. Definitions of Feature Discretization

2.2. Genetic Algorithm Using Binary Coding

2.3. Fitness Function Based on Rough Set

2.4. Reinforcement Learning

2.5. Main Challenges

3. Reinforcement Learning-Based Genetic Algorithm

3.1. Evolution of Discretization Scheme to Be Optimized

3.2. Selection Operator Based on the Control Function

3.3. Crossover Operator Based on Q-Learning

3.4. Mutation Operator Based on Q-Learning

3.5. Flow of RLGA Algorithm

4. Experimental Design

4.1. Data Source

4.2. Configuration of Experimental Environment

4.3. Preparation of Experimental Datasets

5. Results and Discussion

5.1. RLGA versus GA

5.2. RLGA versus Mainstream Discretization Algorithms

5.3. Optimization of Breakpoints on Gaofen-2 Image

6. Conclusion and Future Work

Data Availability

Disclosure

Conflicts of Interest

Acknowledgments

References

Copyright