Abstract
Recently, convolutional neural networks have demonstrated excellent performance on various visual tasks, including the classification of common twodimensional images. In this paper, deep convolutional neural networks are employed to classify hyperspectral images directly in spectral domain. More specifically, the architecture of the proposed classifier contains five layers with weights which are the input layer, the convolutional layer, the max pooling layer, the full connection layer, and the output layer. These five layers are implemented on each spectral signature to discriminate against others. Experimental results based on several hyperspectral image data sets demonstrate that the proposed method can achieve better classification performance than some traditional methods, such as support vector machines and the conventional deep learningbased methods.
1. Introduction
The hyperspectral imagery (HSI) [1] is acquired by remote sensers, which are characterized in hundreds of observation channels with high spectral resolution. Taking advantages of the rich spectral information, numerous traditional classification methods, such as nearestneighbors (nn), minimum distance, and logistic regression [2], have been developed. Recently, some more effective feature extraction methods as well as advanced classifiers were proposed, such as spectralspatial classification [3] and local Fisher discriminant analysis [4]. In the current literatures, support vector machine (SVM) [5, 6] has been viewed as an efficient and stable method for hyperspectral classification tasks, especially for the small training sample sizes. SVM seeks to separate twoclass data by learning an optimal decision hyperplane which best separates the training samples in a kernelincluded highdimensional feature space. Some extensions of SVM in hyperspectral image classification were presented to improve the classification performance [3, 7, 8].
Neural networks (NN), such as multilayer perceptron (MLP) [9] and radial basis function (RBF) [10] neural networks, have already been investigated for classification of remote sensing data. In [11], the authors proposed a semisupervised neural network framework for largescale HSI classification. Actually, in remote sensing classification tasks, SVM is superior to the traditional NN in terms of classification accuracy as well as computational cost. In [12], a deeper architecture of NN has been considered a powerful model for classification, whose classification performance is competitive to SVM.
Deep learningbased methods achieve promising performance in many fields. In deep learning, the convolutional neural networks (CNNs) [12] play a dominant role for processing visualrelated problems. CNNs are biologicallyinspired and multilayer classes of deep learning models that use a single neural network trained end to end from raw image pixel values to classifier outputs. The idea of CNNs was firstly introduced in [13], improved in [14], and refined and simplified in [15, 16]. With the largescale sources of training data and efficient implementation on GPUs, CNNs have recently outperformed some other conventional methods, even human performance [17], on many visionrelated tasks, including image classification [18, 19], object detection [20], scene labeling [21], house number digit classification [22], and face recognition [23]. Besides vision tasks, CNNs have been also applied to other areas, such as speech recognition [24, 25]. The technique has been verified as an effective class of models for understanding visual image content, giving some stateoftheart results on visual image classification and other visualrelated problems. In [26], the authors presented DNN for HSI classification, in which stacked autoencoders (SAEs) were employed to extract discriminative features.
CNNs have been demonstrated to provide even better classification performance than the traditional SVM classifiers [27] and the conventional deep neural networks (DNNs) [18] in visualrelated area. However, since CNNs have been only considered on visualrelated problems, there are rare literatures on the technique with multiple layers for HSI classification. In this paper, we have found that CNNs can be effectively employed to classify hyperspectral data after building appropriate layer architecture. According to our experiments, we observe that the typical CNNs, such as LeNet5 [14] with two convolutional layers, are actually not applicable for hyperspectral data. Alternatively, we present a simple but effective CNN architecture containing five layers with weights for supervised HSI classification. Several experiments demonstrate excellent performance of our proposed method compared to the classic SVM and the conventional deep learning architecture. As far as we know, it is the first time to employ the CNN with multiple layers for HSI classification.
The paper is organized as follows. In Section 2, we give a brief introduction to CNNs. In Section 3, the typical CNN architecture and the corresponding training process are presented. In Section 4, we experimentally compare the performance of our method with SVM and some neural networks with different architectures. Finally, we conclude by summarizing our results in Section 5.
2. CNNs
CNNs represent feedforward neural networks which consist of various combinations of the convolutional layers, max pooling layers, and fully connected layers and exploit spatially local correlation by enforcing a local connectivity pattern between neurons of adjacent layers. Convolutional layers alternate with max pooling layers mimicking the nature of complex and simple cells in mammalian visual cortex [28]. A CNN consists of one or more pairs of convolution and max pooling layers and finally ends with a fully connected neural networks. A typical convolutional network architecture is shown in Figure 1 [24].
In ordinary deep neural networks, a neuron is connected to all neurons in the next layer. CNNs are different from ordinary NN in that neurons in convolutional layer are only sparsely connected to the neurons in the next layer, based on their relative location. That is to say, in a fully connected DNNs, each hidden activation is computed by multiplying the entire input by weights in that layer. However, in CNNs, each hidden activation is computed by multiplying a small local input against the weights . The weights are then shared across the entire input space, as shown in Figure 1. Neurons that belong to the same layer share the same weights. Weight sharing is a critical principle in CNNs since it helps reduce the total number of trainable parameters and leads to more efficient training and more effective model. A convolutional layer is usually followed by a max pooling layer.
Due to the replication of weights in a CNN, a feature may be detected across the input data. If an input image is shifted, the neuron detecting the feature is shifted as much. Pooling is used to make the features invariant from the location, and it summarizes the output of multiple neurons in convolutional layers through a pooling function. Typical pooling function is maximum. A max pooling function basically returns the maximum value from the input. Max pooling partitions the input data into a set of nonoverlapping windows and outputs the maximum value for each subregion and reduces the computational complexity for upper layers and provides a form of translation invariance. To be used for classification, the computation chain of a CNN ends in a fully connected network that integrates information across all locations in all the feature maps of the layer below.
Most of CNNs working in image recognition have the lower layers composed to alternate convolutional and max pooling layers, while the upper layers are fully connected traditional MLP NNs. For example, LeNet5 is such a CNN architecture presented for handwritten digit recognition [14] firstly and then it is successfully used for solving other visualrelated problems. However, LeNet5 might not be directly employed for HSI classification, especially for smallsize data sets, according to our experiments in Section 4. In this paper, we will explore what is the suitable architecture and strategy for CNNbased HSI classification.
3. CNNBased HSI Classification
3.1. Applying CNNs to HSI Classification
The hierarchical architecture of CNNs is gradually proved to be the most efficient and successful way to learn visual representations. The fundamental challenge in such visual tasks is to model the intraclass appearance and shape variation of objects. The hyperspectral data with hundreds of spectral channels can be illustrated as 2D curves (1D array) as shown in Figure 2 (9 classes are selected from the University of Pavia data set). We can see that the curve of each class has its own visual shape which is different from other classes, although it is relatively difficult to distinguish some classes with human eye (e.g., gravel and selfblocking bricks). We know that CNNs can achieve competitive and even better performance than human being in some visual problems, and its capability inspires us to study the possibility of applying CNNs for HSI classification using the spectral signatures.
(a) Asphalt
(b) Meadows
(c) Gravel
(d) Trees
(e) Sheets
(f) Bare soil
(g) Bitumen
(h) Bricks
(i) Shadows
3.2. Architecture of the Proposed CNN Classifier
The CNN varies in how the convolutional and max pooling layers are realized and how the nets are trained. As illustrated in Figure 3, the net contains five layers with weights, including the input layer, the convolutional layer C1, the max pooling layer M2, the full connection layer F3, and the output layer. Assuming represents all the trainable parameters (weight values), and , where is the parameter set between the ()th and the th layer.
In HSI, each HSI pixel sample can be regarded as a 2D image whose height is equal to 1 (as 1D audio inputs in speech recognition). Therefore, the size of the input layer is just , and is the number of bands. The first hidden convolutional layer C1 filters the input data with 20 kernels of size . Layer C1 contains nodes, and . There are trainable parameters between layer C1 and the input layer. The max pooling layer M2 is the second hidden layer, and the kernel size is . Layer M2 contains nodes, and . There is no parameter in this layer. The fully connected layer F3 has nodes and there are trainable parameters between this layer and layer M2. The output layer has nodes, and there are trainable parameters between this layer and layer F3. Therefore, the architecture of our proposed CNN classifier totally has trainable parameters.
Classifying a specified HSI pixel requires the corresponding CNN with the aforementioned parameters, where and are the spectral channel size and the number of output classes of the data set, respectively. In our experiments, is better to be , and . can be any number between 30 and 40, and . is set to be 100. These choices might not be the best but are effective for general HSI data.
In our architecture, layer C1 and M2 can be viewed as a trainable feature extractor to the input HSI data, and layer F3 is a trainable classifier to the feature extractor. The output of subsampling is the actual feature of the original data. In our proposed CNN structure, 20 features can be extracted from each original hyperspectral, and each feature has dimensions.
Our architecture has some similarities to architectures that applied CNN for frequency domain signal in speech recognition [24, 25]. We think it is caused by the similarity between 1D input of speech spectrum and hyperspectral data. Different from [24, 25], our network varies according to the spectral channel size and the number of output classes of input HSI data.
3.3. Training Strategies
Here, we introduce how to learn the parameter space of the proposed CNN classifier. All the trainable parameters in our CNN should be initialized to be a random value between −0.05 and 0.05. The training process contains two steps: forward propagation and back propagation. The forward propagation aims to compute the actual classification result of the input data with current parameters. The back propagation is employed to update the trainable parameters in order to make the discrepancy between the actual classification output and the desired classification output as small as possible.
3.3.1. Forward Propagation
Our ()layer CNN network ( in this work) consists of input units in layer INPUT, output units in layer OUTPUT, and several socalled hidden units in layers C2, M3, and F4. Assuming is the input of the th layer and the output of the ()th layer, then we can compute as where and is a weight matrix of the th layer acting on the input data, and is an additive bias vector for the th layer. is the activation function of the th layer. In our designed architecture, we choose the hyperbolic tangent function as the activation function in layer C1 and layer F3. The maximum function is used in layer M2. Since the proposed CNN classifier is a multiclass classifier, the output of layer F3 is fed to way softmax function which produces a distribution over the class labels, and the softmax regression model is defined as
The output vector of the layer OUTPUT denotes the final probability of all the classes in the current iteration.
3.3.2. Back Propagation
In the back propagation stage, the trainable parameters are updated by using the gradient descent method. It is realized by minimizing a cost function and computing the partial derivative of the cost function with respect to each trainable parameter [29]. The loss function used in this work is defined as where is the number of training samples. is the desired output. is the th value of the actual output (see (3)) of the th training sample and is a vector whose size is . In the desired output of the th sample, the probability value of the labeled class is 1, and the probability values of other classes are 0. means, if is equal to the desired label of the th training sample, its value is 1; otherwise, its value is 0. We add a minus sign to the front of in order to make the computation more convenient.
The derivative of the loss function with respect to is where denotes elementwise multiplication. can be easily represented as Therefore, on each iteration, we would perform the update for adjusting the trainable parameters, where is the learning factor ( in our implementation), and We know that contains and , and where
With an increasing number of training iteration, the return of the cost function is smaller, which indicates that the actual output is closer to the desired output. The iteration stops when the discrepancy between them is small enough. We use average sum of squares to represent the discrepancy. Finally, the trained CNN is ready for HSI classification. The summary of the proposed algorithm is shown in Algorithm 1.

3.4. Classification
Since the architecture and all corresponding trainable parameters are specified, we can build the CNN classifier and reload saved parameters for classifying HSI data. The classification process is just like the forward propagation step, in which we can compute the classification result as (3).
4. Experiments
All the programs are implemented using Python language and Theano [30] library. Theano is a Python library that makes us easily define, optimize, and evaluate mathematical expressions involving multidimensional arrays efficiently and conveniently on GPUs. The results are generated on a PC equipped with an Intel Core i7 with 2.8 GHz and Nvidia GeForce GTX 465 graphics card.
4.1. The Data Sets
Three hyperspectral data, including Indian Pines, Salinas, and University of Pavia scenes, are employed to evaluate the effectiveness of the proposed method. For all the data, we randomly select 200 labeled pixels per class for training and all other pixels in the ground truth map for test. Development data are derived from the available training data by further dividing them into training and testing samples for tuning the parameters of the proposed CNN classifier. Furthermore, each pixel is scaled to uniformly.
The Indian Pines data set was gathered by Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor in northwestern Indiana. There are 220 spectral channels in the 0.4 to 2.45 m region of the visible and infrared spectrum with a spatial resolution of 20 m. From the statistical viewpoint, we discard some classes which only have few labeled samples and select 8 classes for which the numbers of training and testing samples are listed in Table 1. The layer parameters of this data set in the proposed CNN classifier are set as follows: , , , , , , and , and the number of total trainable parameters in the data set is 81408.
The second data employed was also collected by the AVIRIS sensor, capturing an area over Salinas Valley, California, with a spatial resolution of 3.7 m. The image comprises pixels with 220 bands. It mainly contains vegetables, bare soils, and vineyard fields (http://www.ehu.es/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes). There are also 16 different classes, and the numbers of training and testing samples are listed in Table 2. The layer parameters of this data set in our CNN are set to be , , , , , , and , and the number of total trainable parameters in the data set is 82216.
The University of Pavia data set was collected by the Reflective Optics System Imaging Spectrometer (ROSIS) sensor. The image scene, with a spatial coverage of pixels covering the city of Pavia, Italy, was collected under the HySens project managed by DLR (the German Aerospace Agency). The data set has 103 spectral bands prior to water band removal. It has a spectral coverage from 0.43 to 0.86 m and a spatial resolution of 1.3 m. Approximately 42776 labeled pixels with 9 classes are from the ground truth map, and the numbers of training and testing samples are shown in Table 3. The layer parameters of this data set in our CNN are set to be , , , , , , and , and the number of total trainable parameters in the data set is 61249.
4.2. Results and Comparisons
Table 4 provides the comparison of classification performance between the proposed method and the traditional SVM classifier. SVM with RBF kernel is implemented using the libsvm package (http://www.csie.ntu.edu.tw/~cjlin/libsvm); cross validation is also employed to determine the related parameters, and all optimal ones are used in following experiments. It is obvious that our proposed method has better performance (approximate 2% gain) than SVM classifier using all the three data sets. Figures 4, 5, and 6 illustrate the corresponding classification maps obtained with our proposed method and RBFSVM classifier. Furthermore, compared with RBFSVM, the proposed CNN classifier has higher classification accuracy not only for the overall data set but also for almost all the specific classes as shown in Figure 7.
(a)
(b)
(c)
Figure 8 further illustrates the relationship between classification accuracies and the training time (the test time is also included) for three experimental data sets. With the increasing of training time, the classification accuracy of each data can reach over 90%. We must admit that the training process is relatively timeconsuming to achieve good performance; however, the proposed CNN classifier shares the same advantages (e.g., fast on testing) of deep learning algorithms (see Table 5). Moreover, our implementation of CNN could be improved greatly on efficiency, or we can use other CNN frameworks, such as Caffe [31], to reduce training and test time. According to our experiments, it takes only 5 minutes to achieve 90% accuracy on MNIST dataset [32] by using Caffe compared to more than 120 minutes by using our implemented framework.
(a)
(b)
(c)
Figure 9 illustrates the relationship between cost value (see (4)) and the training time for the University of Pavia data set. The value of the loss function is reduced with an increasing number of training iteration, which demonstrates the convergence of our network with only 200 training samples for each class. Moreover, the cost value is still reduced after 5minute training, but the corresponding test accuracy is relatively stable (see Figure 8(a)), which indicates the overfitting problem in this network.
To further verify that the proposed classifier is suitable for classifying data sets with limited training samples, we also compare our CNN with RBFSVM under different training sizes on the University of Pavia data set as shown in Figure 10. It is obvious that our proposed CNN consistently provides higher accuracy than SVM. However, although the conventional deep learningbased method [26] can outperform the SVM classifier, it requires plenty of training samples for constructing autoencoders.
To demonstrate the relationship between classification accuracies and the visual differences of curve shapes (see Figure 2), we present the detailed accuracies of our proposed CNN classifiers for the University of Pavia data set in Table 6. In the table, the cell in the th row, th column means the percentage of the th class samples (according to ground truth) which is classified to the th class. For example, 87.34% of class Asphalt samples are classified correctly, but 6.25% of class Asphalt samples are wrongly classified to class Bitumen. The percentages on diagonal line are just the classification accuracies of corresponding classes. As for one class, the more unique the corresponding curve shape is, the higher accuracy the proposed CNN classifier can achieve (check the class Shadow and class Sheets in Figure 2 and Table 6). The more similar two curves are, the higher opportunity they are wrongly classified to each other (check the class Gravel and class Bricks in Figure 2 and Table 6). Furthermore, the excellent performance verifies that the proposed CNN classifier has discriminative capability to extract subtle visual features, which is even superior to human vision for classifying complex curve shapes.
Finally, we also implement three other types of neural network architectures for the Indian Pines data set using the same training and test samples. The first one is a simple architecture with only two fully connected layers beside the input layer. The second one is LeNet5 which is a classic CNN architecture with two convolutional layers. The third one is a conventional deep neural networks (DNNs) with 3 hidden fully connected layers (a 2206040208 architecture as suggested in [26]). The classification performance is summarized in Table 5. From Table 5, we can see that our CNN classifier achieves the highest accuracy with competitive training and testing computational cost. LeNet5 and DNNs cost more time to train models due to their complex architecture, but limited training samples restrict their capabilities of classification (only 20% samples selected for testing in [26] compared with 95% in our experiment). Another reason for the difficulty that deeper CNNs and DNNs face to achieve higher accuracies could be that the HSI lacks the type of high frequency signal commonly seen in the computer vision domain (see Figure 2).
5. Conclusion and Future Work
In this paper, we proposed a novel CNNbased method for HSI classification, inspired by our observation that HSI classification can be implemented via human vision. Compared with SVMbased classifier and conventional DNNbased classifier, the proposed method could achieve higher accuracy using all the experimental data sets, even with a small number of training samples.
Our work is an exploration of using CNNs for HSI classification and has excellent performance. The architecture of our proposed CNN classifier only contains one convolutional layer and one fully connected layer, due to the small number of training samples. In the future, a network architecture called a Siamese Network [33] might be used, which has been proved to be robust in the situation where the number of training samples per category is small. Some techniques, such as Dropout [34], can also be used to alleviate the overfitting problem caused by limited training samples. Furthermore, recent researches in deep learning have indicated that unsupervised learning can be employed to train CNNs, reducing the requirement of labeled samples significantly. Deep learning, especially deep CNNs, should have great potentiality for HSI classification in the future. Moreover, in the current work, we do not consider the spatial correlation and only concentrate on the spectral signatures. We believe that some spatialspectral techniques also can be applied to further improve the CNNbased classification. At last, we plan to employ efficient deep CNN frameworks, such as Caffe, to improve our computing performance.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work was supported jointly by the National Natural Science Foundation of China (nos. 61371165 and 61302164), the 973 Program of China (no. 2011CB706900), the Program for New Century Excellent Talents in University under Grant no. NCET110711, and the Interdisciplinary Research Project in Beijing University of Chemical Technology. Wei Hu and Fan Zhang are also supported by the Beijing Higher Education Young Elite Teacher Project under Grant nos. YETP0501 and YETP0500, respectively.