- About this Journal
- Abstracting and Indexing
- Aims and Scope
- Annual Issues
- Article Processing Charges
- Articles in Press
- Author Guidelines
- Bibliographic Information
- Citations to this Journal
- Contact Information
- Editorial Board
- Editorial Workflow
- Free eTOC Alerts
- Publication Ethics
- Reviewers Acknowledgment
- Submit a Manuscript
- Subscription Information
- Table of Contents
BioMed Research International
Volume 2013 (2013), Article ID 239628, 13 pages
Recognition of Multiple Imbalanced Cancer Types Based on DNA Microarray Data Using Ensemble Classifiers
1School of Computer Science and Engineering, Jiangsu University of Science and Technology, No. 2 Mengxi Road, Zhenjiang 212003, China
2Department of Radiology, Carver College of Medicine, The University of Iowa, Iowa City, IA 52242, USA
3School of Biology and Chemical Engineering, Jiangsu University of Science and Technology, No. 2 Mengxi Road, Zhenjiang 212003, China
Received 7 April 2013; Revised 8 July 2013; Accepted 17 July 2013
Academic Editor: Alexander Zelikovsky
Copyright © 2013 Hualong Yu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
DNA microarray technology can measure the activities of tens of thousands of genes simultaneously, which provides an efficient way to diagnose cancer at the molecular level. Although this strategy has attracted significant research attention, most studies neglect an important problem, namely, that most DNA microarray datasets are skewed, which causes traditional learning algorithms to produce inaccurate results. Some studies have considered this problem, yet they merely focus on binary-class problem. In this paper, we dealt with multiclass imbalanced classification problem, as encountered in cancer DNA microarray, by using ensemble learning. We utilized one-against-all coding strategy to transform multiclass to multiple binary classes, each of them carrying out feature subspace, which is an evolving version of random subspace that generates multiple diverse training subsets. Next, we introduced one of two different correction technologies, namely, decision threshold adjustment or random undersampling, into each training subset to alleviate the damage of class imbalance. Specifically, support vector machine was used as base classifier, and a novel voting rule called counter voting was presented for making a final decision. Experimental results on eight skewed multiclass cancer microarray datasets indicate that unlike many traditional classification approaches, our methods are insensitive to class imbalance.
Microarray technology allows large-scale and parallel measurements for expression of around thousands, perhaps even tens of thousands, of genes. It has been one of the most successful molecular biology technologies in the postgenome era and has been widely applied to predict gene functions , provide invaluable information for drug discovery [2, 3], investigate gene regulatory mechanisms [4, 5], find new subtypes of a specific tumor [6, 7], and classify cancers [8, 9]. Among these applications, cancer classification, which has been the subject of extensive research all around the world, is most promising . However, microarray data are known to have some features, such as high dimension, small sample, high noise, high redundancy, and skewed class distribution which is called class imbalance problem. Class imbalance occurs when examples from one class outnumber those of the other class, which results in great underestimation of the classification performance of the minority, thereby further affecting the evaluation precision of the overall classification performance. In other words, developing a clinical tumor diagnostic system is meaningless if class imbalance is not considered.
Recent studies have addressed this problem in the context of cancer classification based on microarray data [11–18]. Unfortunately, most existing work has only considered binary-class imbalance and ignored the multiclass problem, that is, identifying multiple imbalanced tumor types or several skewed subtypes of a special tumor. Applying traditional supervised learning algorithms that solve minimum classification errors will provide inaccurate classification results. Furthermore, addressing skewed multiclass problems is more difficult than dealing with binary-class imbalance problems .
Generally speaking, support vector machine (SVM) is the best choice for classifying cancer microarray data because of its advantages, such as its high generalization capability, absence of local minima, and adaptability for high-dimension and small sample data . However, SVM was initially designed for binary-class problems. Therefore, to apply SVM to multiclass problems, it should be reconfigured for multiple binary-class problems by using a coding strategy . Previous studies have presented several well-known coding strategies, including one-against-all (OAA), one-against-one (OAO), decision directed acyclic graph (DDAG), and error correcting output codes (ECOC). These strategies have also been used to classify multiclass cancer microarray data [22–24]. Statnikov et al.  systematically assessed these strategies by performing experiments and found that OAA often produces better classification accuracy. In the present study, we use OAA as a baseline coding strategy. We also note that this decomposition can further damage the equilibrium of training instances. Therefore, one approach for effective class imbalance correction should be carried out in each binary-class branch.
In this paper, we attempted to address the multiclass imbalance classification problem of cancer microarray data by using ensemble learning. Ensemble learning has been used to improve the accuracy of feature gene selection  and cancer classification [27–29]. First, our method used OAA coding to divide multiclass problems into multiple binary-class problems. Next, we designed an improved random subspace generation approach called feature subspace (FSS) to produce a large number of accurate and diverse training subsets. We then introduced one of two correction technologies, namely, either decision threshold adjustment (THR)  or random undersampling (RUS) , into each training subset to deal with class imbalance. Finally, a novel voting rule based on counter voting was presented, which made the final decision in ensemble learning. We evaluated the proposed method by using eight multiclass cancer DNA microarray datasets that have different numbers of classes, genes, and samples, as well as class imbalance ratios. The experimental results demonstrated that the proposed method outperforms many traditional classification approaches because it produces more balanced and robust classification results.
The rest of this paper is organized as follows. In Section 2, the methods referred to in this study are introduced in detail. Section 3 briefly describes the datasets that were used. Section 4 introduces performance evaluation metrics and experimental settings. Results and discussions are presented in Section 5. Section 6 summarizes the main contributions of this paper.
2.1. Coding Strategies for Transforming Multiclass into Multiple Binary Classes
Coding strategies are often used to transform multiclass into multiple binary-classes . OAA, OAO, and ECOC can be described by a code matrix , where each row contains a code word assigned to each class, and each column defines a binary partition of classes. Specifically, we assign +1, −1, or 0 for each element in . An element with +1 value indicates that the th class is labeled as positive for the th binary classifier, −1 represents that the th class in the th binary classifier is labeled as negative, and 0 means that the th class does not participate in the induction of the th classifier.
Without loss of generality, a problem of four classes is assumed; that is, . OAA generates classifiers in which each one is trained to distinguish a class from the remaining classes. The code matrix of OAA is presented in Figure 1(a). In practical applications, OAA assigns the class label with the highest decision output value to the test instance. Unlike OAA, OAO trains binary classifiers and assigns each one by using only two original classes and simply ignoring the others. Its code matrix is shown in Figure 1(b). The decoding rule of OAO is majority voting; that is, the test instance is designated to the class with the most votes. ECOC proposed by Dietterich and Bariki  uses error correcting codes to denote classes of a multiclass problem. For each column of the code matrix, one or several classes are denoted as positive, and the remainder is designated as negative. In ECOC, hamming distance is applied as decoding strategy. In particular, when using an exhaustive code to construct the code matrix of ECOC, it can generate more binary classifiers than OAA and OAO. The size of ECOC is , and its code matrix is described in Figure 1(c).
DDAG  has the same coding rule as OAO but uses a totally different decoding strategy. It organizes all binary classifiers into one hierarchical structure (see Figure 1(d)) and makes a decision for test samples from root to leaf, which is helpful for decreasing time complexity of the testing process.
To our knowledge, no previous work has considered the effect of class imbalance on these coding strategies, although some have indicated that it is, in fact, harmful [33, 34]. In this paper, we proposed two solutions for this problem and used OAA coding as the baseline.
2.2. Feature Subspace Generation Technology
The performance of ensemble learning is related to two factors: accuracy and diversity of base classifiers . The generalization error of ensemble learning can be calculated by using the following equation: where and are averages of generalization errors and diversities, respectively. Therefore, to create a successful ensemble learning model, two factors should be considered simultaneously. The more accurate each base classifier and the more diverse different base classifiers, the better the classification performance of the ensemble learning. However, these two factors are conflicting; that is, with the increase in average accuracy, the average diversity inevitably declines, and vice versa. Many effective ensemble learning methods are available, including Bagging , AdaBoost , random subspace  and random forest . However, we have observed that these methods are not sufficiently effective in classifying high-dimensional data. Therefore, we used a modified random subspace method , and proposed an FSS generation strategy, which is described below.
DNA microarray data are known to contain numerous noisy and redundant genes, which can negatively affect classification performance and should thus be preliminarily eliminated. FSS generation strategy uses hierarchical clustering, which uses Pearson correlation coefficient (PCC) as a similarity measure to delete redundant genes and signal-to-noise ratio (SNR) feature selection method  to remove noisy genes. PCC evaluates the similarity between two genes and by using the following equation: where is the expression value of the gene on the th sample, represents the mean value of , and denotes the number of training samples. A larger PCC between two genes indicates that the genes have greater similarity. Using this method ensures that all genes could be grouped into clusters, where is the number of clusters. Obviously, redundant genes can emerge in the same clusters. For this, we use the SNR feature selection method  to select differentially expressed genes in each cluster, with the computational formula listed as follows: where and are mean values of gene in positive class and negative class, and and are their standard deviations, respectively. The extracted features are clearly closely correlated with the classification task without being redundant with each other. We call the space that merely contains the extracted genes the feature space from which multiple feature subspaces can be generated. If the dimension of feature subspace is , where , then a feature subspace can be generated by using the following random project function:
By using the random project function , we can repeatedly produce multiple diverse feature subspaces. For a given high-dimensional training set , the pseudocode description of the FSS generalization algorithm is presented in Pseudocode 1.
We also analyze the reason behind the ability of FSS to promote equilibrium relationship between accuracy and diversity of base classifiers. Suppose is one gene in feature space that has been integrated into feature subspace . Then the probability that has simultaneously appeared in another feature subspace is
This equation means that for any two feature subspaces, their coselection rate is, in theory, about . Moreover, because any two genes in the feature space can be regarded as approximatively nonredundant, the theoretical diversity between two feature subspaces div can be computed by the following:
When is much larger than , diversity among the feature subspaces can be guaranteed. is an important parameter that influences the accuracy of base classifiers and should not be assigned an overly small value. In addition, a constructed ensemble learning model theoretically has different combinations, such that the number of different combinations is deduced to reach its peak value when .
2.3. Support Vector Machine and Its Correction Technologies for Class Imbalance Problem
SVM, which is based on the structural risk minimization theory, is one of the most popular classification algorithms. The decision function of SVM is listed as follows: where represents the number of support vectors, is the Lagrange multiplier, is the bias of optimum classification hyperplane, and denotes the kernel function. Some previous studies have found that the radial basis kernel function (RBF) generally produces better classification accuracy than many other kernel functions [20, 30]. RBF kernel is presented as where is the parameter that indicates the width of the RBF kernel.
Although SVM is more robust to class imbalance than many other machine learning methods because its classification hyperplane only associates with a few support vectors, it can still be, more or less, affected by skewed class distribution. Previous studies [40, 41] have found that the classification hyperplane can be pushed toward the minority class if the classification data is skewed (see Figure 2(a)).
Class imbalance correction technologies of SVM can be roughly divided into three categories: sampling [30, 40], weighting [41, 42], and decision threshold adjustment , that is, threshold moving. Sampling is the most direct solution for class imbalance. It increases instances of minority class  or decreases examples of majority class  to mediate the skewed scaling relation. The former is called oversampling and the latter is called undersampling. Weighting , which is also known as cost-sensitive learning, assigns different penalty factors for the samples of positive and negative classes. Generally speaking, the penalty factor of positive class is much larger than that of negative class . Phoungphol et al.  used ramp loss function to construct a more robust and cost-sensitive support vector machine (Ramp-MCSVM) and used it to classify multiclass imbalanced biomedical data. Decision threshold adjustment based on support vector machine (SVM-THR) directly pushes classification hyperplane toward the majority class. Lin and Chen  suggested adopting SVM-THR to classify severely imbalanced bioinformatics data.
In this paper, to reduce time complexity, we used SVM based on random undersampling (SVM-RUS)  (see Figure 2(b)) and SVM with decision threshold adjustment (SVM-THR)  (see Figure 2(c)) to deal with class imbalance problem. The decision threshold is adjusted by using the following default equation : where and are the number of examples that belong to the positive class and the negative class, respectively. For one test sample , supposing that the original decision function is , the adjusted decision function can be represented as .
2.4. Ensemble Learning Framework Based on Feature Subspace and Counter Voting Integration Rule for Classifying Imbalanced Multiclass Cancer Microarray Data
Ensemble learning often provides a framework to generate multiple weak classifiers and aggregates these by using an integration rule to become a strong classifier. The integration rules mainly include majority voting and weighted voting. With the characteristics of multiclass problem taken into consideration and referring to the idea of majority voting, we propose a novel integration rule called counter voting. For each decomposed binary-class branch in OAA, one counter is assigned, which indicates the proportion of test sample that belongs to the corresponding positive class. All counters compete with each other to select the category of the test sample by using the following equation:
The pseudo-code description and graphical representation of our proposed ensemble learning algorithms are given in Pseudocode 2 and Figure 3, respectively. We call these algorithms as EnSVM-OAA(THR) and EnSVM-OAA(RUS). Figure 3 shows that if one classification task is binary, counter voting turns into majority voting. Counter voting, rather than majority voting or weighted voting, is used to classify multiclass data because generating feature space on each binary-class is more accurate than directly generating feature space on multiple classes. Our proposed ensemble learning framework also has the same time complexity as aggregating SVM-OAAs by using majority voting.
Eight skewed multiclass cancer microarray datasets [6, 7, 43–48] were used to verify the effect of our proposed ensemble learning methods, which have 3 to 26 classes, 50 to 308 instances, 2308 to 15009 genes, and imbalance ratios in the range of 2.14 to 23.17. These datasets are available at http://www.gems-system.org/, and detailed information about these data is shown in Table 1.
4. Performance Evaluation Metrics and Experimental Settings
When one classification task is skewed, the overall classification accuracy Acc is no longer an appropriate evaluation metric for estimating the quality of a classifier. In this case, a confusion matrix described in Table 2 is usually employed.
The description in Table 2 gives four baseline statistical components, where TP and FN denote the number of positive examples which are accurately and falsely predicted, respectively, and TN and FP represent the number of negative samples that are predicted accurately and wrongly, respectively. Two frequently used measures for class imbalance problem, namely, -measure and -mean, can be regarded as functions of these four statistical components and are calculated as follows: where Precision, Recall, TPR, and TNR can be further defined as follows: The overall classification accuracy Acc can be calculated by using the following equation:
However, these evaluation metrics are merely appropriate for estimating binary-class imbalance tasks. To extend these metrics to multiclass, some transformations should be considered. -mean computes the geometric mean of all classes’ accuracies and is described as follows: where denotes the accuracy of the th class. -measure can be transformed as -score , which can be calculated by using the following formula: where can be calculated further by using the following equation: and the Acc metric can also be transformed as follows: where is the percentage of samples in the th class.
To impartially and comprehensively assess the classification performance, we use three extended measures, namely, -mean,-score, and Acc, which are described in (14), (15), and (17), respectively, as evaluation metrics.
We empirically performed threefold cross-validation  to evaluate classification performance. Considering the randomness of the sample set partition, each experiment was randomly repeated 10 times. The final values of Acc, -score, and -mean were averaged by these 10 runs. The penalty factor and the width parameter of RBF kernel function were tuned by using grid search with threefold cross-validation, where and . In addition, the initial dimension of feature space and that of feature subspace are empirically assigned as 100 and 20, respectively. , which indicates the number of base classifiers in each OAA branch, is also empirically assigned as 100.
To demonstrate the advantage of our methods, we evaluated them in comparison with 10 other classification methods, namely, SVM-OAA, SVM-OAO, SVM-DDAG, SVM-ECOC, single SVM-OAA classifier with THR and RUS correction strategies (OAA-SVM(THR) and OAA-SVM(RUS)), ensemble of SVM-OAA without considering class imbalance (EnSVM-OAA), MCSVM , Ramp-MCSVM , and AdaBoost.NC . To equitably compare the performance of various methods, we used the same common parameters. The other parameters used were the default ones found in references [19, 41, 42].
5. Results and Discussions
The experimental results of 12 classification algorithms on 8 datasets are reported in Tables 3, 4 and 5, where the best result in each dataset is highlighted in bold, the second best is underlined, and the worst is italicized. From Tables 3 to 5, we observe the following.(i)SVM with various coding strategies exhibits quite similar classification performance in terms of Acc, -score, and -mean evaluation metrics. Compared with its three competitors, SVM-OAA does not show sufficient superiority, although it simplifies transformation by decomposing each multiclass problem to the least binary-class problems. In addition, we found that all four traditional classification algorithms are sensitive to class imbalance.(ii)Some datasets are sensitive to class imbalance but others are not, as shown by the difference between Acc and -mean values. An Acc value that is much larger than the -mean value means that the corresponding classifier is significantly affected by imbalanced class distribution, which was observed in several datasets used in the study, including Brain_Tumor1, 11_Tumors, and 14_Tumors. Brain_Tumor2 and Lung_Cancer were both slightly sensitive to class imbalance as well. We consider these results to be related to a weighted combination of number of classes, class imbalance ratio, and class overlapping, as explained by previous studies [19, 50, 51].(iii)Both THR and RUS correction technologies help SVM-OAA classifier promote classification performance on those sensitive datasets. The promotions are better reflected by the -score and -mean metrics, which are used to evaluate the balance level of classification results. Thus, the correction technologies are useless when the classification tasks are robust to class imbalance.(iv)In contrast with SVM-OAA, the ensemble version EnSVM-OAA helps to slightly improve the overall classification accuracy Acc, with possible sacrifice of two other evaluation metrics on most datasets, which means that classification accuracies between majority and minority classes are further increased.(v)Our proposed algorithms outperform other classification algorithms, including several subtle multiclass imbalance classification algorithms [19, 41, 42], in terms of all evaluation criteria for most datasets and especially on the sensitive ones. During the experiments, we observed an interesting phenomenon: EnSVM-OAA(RUS) generally has more stable performance than its partner, although EnSVM-OAA(THR) produces slightly better recognition results on several datasets. We consider that the excessive threshold adjustment negatively affects the recognition accuracy of majority classes to a large extent, which further affects overall prediction accuracy. In practical applications, the decision threshold adjustment function should be subtly designed by considering real distribution of instances.
The classification performance of our proposed algorithms is restricted by many factors, including the size of feature space, the size of feature subspace, and the number of base classifiers; the size of feature subspace is the most significant factor. To clarify its influence mechanism, we designed a group of new experiments in which the dimension of feature subspace is assigned as 10, 20, 30, 50, and 80. The other parameters follow the initial settings in Section 4. The average results of 10 random runs for EnSVM-OAA(THR) and EnSVM-OAA(RUS) are reported in Figures 4 and 5, respectively.
Although some fluctuations were observed, Figures 4 and 5 nonetheless reveal a common trend that optimal performances often emerge with a feature subspace of 10 to 30 dimensions. With the further increase of the feature subspace dimension, the classification performance drops rapidly, which indicates that selecting a feature subspace with 10 to 30 dimensions can maximize the balanced relationship between accuracy and diversity of base classifiers. This result can be easily explained by the following: extracting a too-small subgroup of feature genes can negatively affect the performance of each base classifier, whereas using too many feature genes can negatively affect diversity among base classifiers. In fact, in practical applications, the optimal dimension can be determined through internal multiple-fold cross-validation of the training sets. The experimental results help guide the construction of the optimal classification model.
In this paper, we attempted to address multiclass imbalanced classification problem in tumor DNA microarray data by using ensemble learning. The proposed solution contributes in three ways: (1) an improved version of random subspace called feature subspace, which is specifically designed for high-dimensional classification tasks, is proposed to promote a balanced relationship between accuracy and diversity of base classifiers in ensemble learning; (2) two simple correction technologies are adopted in each branch of OAA to alleviate the effect of class imbalance; and (3) a novel ensemble integration strategy called counter voting, which is based on majority voting, is presented to output the final class label. The empirical results show that our proposed classification algorithms outperform many traditional classification approaches and yield more balanced and robust classification results.
Our goal is for the proposed algorithms to be applied in real clinical cancer diagnostic systems based on DNA microarray data in the future. Our future work will consider the extension of correction strategies and classification approaches to deal with this problem and will also explore some efficient solutions with several other coding strategies.
This work is partially supported by the National Natural Science Foundation of China under Grant no. 61100116, the Nature Science Foundation of the Jiangsu Higher Education Institutes of China under Grant no. 12KJB520003, and the Natural Science Foundation of Jiangsu Province of China (BK2011492).
- T. Puelma, R. A. Gutierrez, and A. Soto, “Discriminative local subspaces in gene expression data for effective gene function prediction,” Bioinformatics, vol. 28, no. 17, pp. 2256–2264, 2012.
- F. De Longueville, D. Surry, G. Meneses-Lorente et al., “Gene expression profiling of drug metabolism and toxicology markers using a low-density DNA microarray,” Biochemical Pharmacology, vol. 64, no. 1, pp. 137–149, 2002.
- S. Bates, “The role of gene expression profiling in drug discovery,” Current Opinion in Pharmacology, vol. 11, no. 5, pp. 549–556, 2011.
- M. Kabir, N. Noman, and H. Iba, “Reverse engineering gene regulatory network from microarray data using linear time-variant model,” BMC Bioinformatics, vol. 11, no. 1, article S56, 2010.
- G. Chalancon, C. N. J. Ravarani, S. Balaji et al., “Interplay between gene expression noise and regulatory network architecture,” Trends in Genetics, vol. 28, no. 5, pp. 221–232, 2012.
- T. R. Golub, D. K. Slonim, P. Tamayo et al., “Molecular classification of cancer: class discovery and class prediction by gene expression monitoring,” Science, vol. 286, no. 5439, pp. 531–527, 1999.
- C. L. Nutt, D. R. Mani, R. A. Betensky et al., “Gene expression-based classification of malignant gliomas correlates better with survival than histological classification,” Cancer Research, vol. 63, no. 7, pp. 1602–1607, 2003.
- X. Wang and R. Simon, “Microarray-based cancer prediction using single genes,” BMC Bioinformatics, vol. 12, article 391, 2011.
- S. Ghorai, A. Mukherjee, S. Sengupta, and P. K. Dutta, “Cancer classification from gene expression data by NPPC ensemble,” IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 8, no. 3, pp. 659–671, 2011.
- T. D. Pham, C. Wells, and D. I. Grane, “Analysis of microarray gene expression data,” Current Bioinformatics, vol. 1, no. 1, pp. 37–53, 2006.
- K. Yang, Z. Cai, J. Li, and G. Lin, “A stable gene selection in microarray data analysis,” BMC Bioinformatics, vol. 7, article 228, 2006.
- G.-Z. Li, H.-H. Meng, and J. Ni, “Embedded gene selection for imbalanced microarray data analysis,” in Proceedings of the 3rd International Multi-Symposiums on Computer and Computational Sciences (IMSCCS '08), pp. 17–24, Shanghai, China, October 2008.
- A. H. M. Kamal, X. Zhu, and R. Narayanan, “Gene selection for microarray expression data with imbalanced sample distributions,” in Proceedings of the International Joint Conference on Bioinformatics, Systems Biology and Intelligent Computing (IJCBS '09), pp. 3–9, Shanghai, China, August 2009.
- R. Blagus and L. Lusa, “Class prediction for high-dimensional class-imbalanced data,” BMC Bioinformatics, vol. 11, article 523, 2010.
- M. Wasikowski and X.-W. Chen, “Combating the small sample class imbalance problem using feature selection,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1388–1400, 2010.
- H. L. Yu, J. Ni, and J. Zhao, “ACOSampling: an ant colony optimization-based undersampling method for classifying imbalanced DNA microarray data,” Neurocomputing, vol. 101, no. 2, pp. 309–318, 2013.
- W. J. Lin and J. J. Chen, “Class-imbalanced classifiers for high-dimensional data,” Briefings in Bioinformatics, vol. 14, no. 1, pp. 13–26, 2013.
- R. Blagus and L. Lusa, “Evaluation of SMOTE for high-dimensional class-imbalanced microarray data,” in Proceedings of the 11th International Conference on Machine Learning and Applications, pp. 89–94, Boca Raton, Fla, USA, 2012.
- S. Wang and X. Yao, “Multiclass imbalance problems: analysis and potential solutions,” IEEE Transactions on Systems, Man, and Cybernetics B, vol. 42, no. 4, pp. 1119–1130, 2012.
- M. J. Abdi, S. M. Hosseini, and M. Rezghi, “A novel weighted support vector machine based on particle swarm optimization for gene selection and tumor classification,” Computational and Mathematical Methods in Medicine, vol. 2012, Article ID 320698, 7 pages, 2012.
- A. C. Lorena, A. C. P. L. F. De Carvalho, and J. M. P. Gama, “A review on the combination of binary classifiers in multiclass problems,” Artificial Intelligence Review, vol. 30, no. 1–4, pp. 19–37, 2008.
- C.-H. Yeang, S. Ramaswamy, P. Tamayo et al., “Molecular classification of multiple tumor types,” Bioinformatics, vol. 17, supplement 1, pp. S316–S322, 2001.
- L. Shen and E. C. Tan, “Reducing multiclass cancer classification to binary by output coding and SVM,” Computational Biology and Chemistry, vol. 30, no. 1, pp. 63–71, 2006.
- S. J. Joseph, K. R. Robbins, W. Zhang, and R. Rekaya, “Comparison of two output-coding strategies for multi-class tumor classification using gene expression data and latent variable model as binary classifier,” Cancer Informatics, vol. 9, pp. 39–48, 2010.
- A. Statnikov, C. F. Aliferis, I. Tsamardinos, D. Hardin, and S. Levy, “A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis,” Bioinformatics, vol. 21, no. 5, pp. 631–643, 2005.
- L. Nanni, S. Brahnam, and A. Lumini, “Combining multiple approaches for gene microarray classification,” Bioinformatics, vol. 28, no. 8, pp. 1151–1157, 2012.
- A. Bertoni, R. Folgieri, and G. Valentini, “Classification of DNA microarray data with Random Projection Ensembles of Polynomial SVMs,” in Proceedings of the 18th Italian Workshop on Neural Networks, pp. 60–66, Vietri sul Mare, Italy, 2008.
- Y. Chen and Y. Zhao, “A novel ensemble of classifiers for microarray data classification,” Applied Soft Computing Journal, vol. 8, no. 4, pp. 1664–1669, 2008.
- K.-J. Kim and S.-B. Cho, “An evolutionary algorithm approach to optimal ensemble classifiers for DNA microarray data analysis,” IEEE Transactions on Evolutionary Computation, vol. 12, no. 3, pp. 377–388, 2008.
- A. Anand, G. Pugalenthi, G. B. Fogel, and P. N. Suganthan, “An approach for classification of highly imbalanced data using weighting and undersampling,” Amino Acids, vol. 39, no. 5, pp. 1385–1391, 2010.
- T. G. Dietterich and G. Bariki, “Solving multiclass learning problems via error-correcting output codes,” Journal of Artificial Intelligence Research, vol. 2, pp. 263–286, 1995.
- B. Kijsirikul and N. Ussivakul, “Multiclass support vector machines using adaptive directed acyclic graph,” in Proceedings of the International Joint Conference on Neural Networks (IJCNN '02), pp. 980–985, Honolulu, Hawaii, USA, May 2002.
- H. L. Yu, S. Gao, B. Qin, et al., “Multiclass microarray data classification based on confidence evaluation,” Genetics and Molecular Research, vol. 11, no. 2, pp. 1357–1369, 2012.
- J.-H. Hong and S.-B. Cho, “A probabilistic multi-class strategy of one-vs.-rest support vector machines for cancer classification,” Neurocomputing, vol. 71, no. 16–18, pp. 3275–3281, 2008.
- A. Krogh and J. Vedelsby, “Neural network ensembles, cross validation, and active learning,” Advances in Neural Information Processing Systems, vol. 7, pp. 231–238, 1995.
- L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, no. 2, pp. 123–140, 1996.
- X. Li, L. Wang, and E. Sung, “AdaBoost with SVM-based component classifiers,” Engineering Applications of Artificial Intelligence, vol. 21, no. 5, pp. 785–795, 2008.
- T. K. Ho, “The random subspace method for constructing decision forests,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 832–844, 1998.
- L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.
- M. Gao, X. Hong, S. Chen, and C. J. Harris, “A combined SMOTE and PSO based RBF classifier for two-class imbalanced problems,” Neurocomputing, vol. 74, no. 17, pp. 3456–3466, 2011.
- R. Akbani, S. Kwek, and N. Japkowicz, “Applying support vector machines to imbalanced datasets,” in Proceedings of the 15th European Conference on Machine Learning (ECML '04), vol. 3201 of Lecture Notes in Computer Science, pp. 39–50, September 2004.
- P. Phoungphol, Y. Zhang, and Y. Zhao, “Robust multiclass classification for learning from imbalanced biomedical data,” Tsinghua Science and Technology, vol. 17, no. 6, pp. 619–628, 2012.
- S. A. Armstrong, J. E. Staunton, L. B. Silverman et al., “MLL translocations specify a distinct gene expression profile that distinguishes a unique leukemia,” Nature Genetics, vol. 30, no. 1, pp. 41–47, 2002.
- A. I. Su, J. B. Welsh, L. M. Sapinoso et al., “Molecular classification of human carcinomas by use of gene expression signatures,” Cancer Research, vol. 61, no. 20, pp. 7388–7393, 2001.
- J. Khan, J. S. Wei, M. Ringnér, et al., “Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks,” Nature Medicine, vol. 7, no. 6, pp. 673–679, 2001.
- S. L. Pomeroy, P. Tamayo, M. Gaasenbeek et al., “Prediction of central nervous system embryonal tumour outcome based on gene expression,” Nature, vol. 415, no. 6870, pp. 436–442, 2002.
- A. Bhattacharjee, W. G. Richards, J. Staunton et al., “Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses,” Proceedings of the National Academy of Sciences of the United States of America, vol. 98, no. 24, pp. 13790–13795, 2001.
- S. Ramaswamy, P. Tamayo, R. Rifkin et al., “Multiclass cancer diagnosis using tumor gene expression signatures,” Proceedings of the National Academy of Sciences of the United States of America, vol. 98, no. 26, pp. 15149–15154, 2001.
- A. Özgür, L. Özgür, and T. Güngör, “Text categorization with class-based and corpus-based keyword selection,” Lecture Notes in Computer Science, vol. 3733, pp. 606–615, 2005.
- N. Japkowicz and S. Stephen, “The class imbalance problem: a systematic study,” Intelligent Data Analysis, vol. 6, no. 5, pp. 429–449, 2002.
- X.-Y. Liu, J. Wu, and Z.-H. Zhou, “Exploratory undersampling for class-imbalance learning,” IEEE Transactions on Systems, Man, and Cybernetics B, vol. 39, no. 2, pp. 539–550, 2009.