Research Article | Open Access
Ground-Based Cloud-Type Recognition Using Manifold Kernel Sparse Coding and Dictionary Learning
Recognizing cloud type of ground-based images automatically has a great influence on the weather service but poses a significant challenge. Based on the symmetric positive definite (SPD) matrix manifold, a novel method named “manifold kernel sparse coding and dictionary learning” (MKSCDL) is proposed for cloud classification. Different from classical features extracted in the Euclidean space, the SPD matrix fuses multiple features and represents non-Euclidean geometric characteristics. MKSCDL is composed of three steps: feature extraction, dictionary learning, and classification. With the learned dictionary, the SPD matrix of the cloud image can be described with the sparse code. The experiments are conducted on two different ground-based cloud image datasets. Benefitting from the sparse representation on the Riemannian matrix manifold, compared to the recent baselines, experimental results demonstrate that MKSCDL possesses a more competitive performance on both grayscale and colour image datasets.
Clouds play an essential role in the circulation of water vapour and affects the earth’s energy balance [1–3]. In the study of weather forecasting and climate change, clouds are always regarded as the core factor . The traditional cloud observation is much dependent on the observers’ experience, and thus, it is time-consuming. The substantial and noticeable development of hardware and digital imaging techniques makes it possible to observe the cloud automatically and continuously. Compared with satellite images, the ground-based images hold a high spatial resolution at a local scale .
Attributing to the application of sky imagers and ceilometer, automatic observation has been realized in cloud cover and cloud base height . To identify the cloud type accurately and effectively, many attempts have recently been made to address this challenging issue [5–17]. Buch et al.  adopted texture, position, and pixel brightness features together from the whole-sky images and classified the data with decision trees. To test the texture feature extraction approach, autocorrelation, cooccurrence matrices, edge frequency, Law’s features, and primitive length were applied for cloud recognition . Calbó and Sabburg  presented statistical texture features, Fourier transform features, and thresholded image features to identify the images taken by the whole-sky imager (WSI) and total sky imager (TSI). Heinle et al.  extracted 12 statistical features to represent the image’s colour and texture and then employed the k-nearest-neighbour (KNN) classifier to distinguish seven different sky conditions. As a clear distinction exists in texture orientation between the satellite images and the ground-based images, Gabor-based multiple features were utilized for classification with support vector machine (SVM) and achieved an overall accuracy of 88.3% . Different from the typical local binary patterns (LBPs), weighted local binary patterns (WLBPs)  were presented by fusing the variance of a local patch to enhance the contrast for recognizing cloud types. Cheng and Yu  combined statistical features and LBPs with the Bayesian classifier to perform block-based classification (BC). In addition to texture features, Liu et al.  employed 7 structure features from the edge image to describe the structure characteristic of the infrared clouds. Zhuo et al.  indicated that simply using texture or structure features separately may not produce excellent classification performance. Hence, both texture and structure features were captured to obtain the cloud type with SVM. Furthermore, Xia et al.  and Xiao et al.  proposed to make use of multiple features together, including colour, texture, and structure features for cloud-type recognition, and the experiments validated that the integration of various features performed better than other cases. The physical features are also of great importance to represent clouds. Kazantzidis et al.  introduced the solar zenithal angle, the total cloud coverage, the visible percentage of sun, and the existence of rain in sky images to describe the physical property. Besides 12 image features extracted from the sky camera image, Tato et al.  combined another 7 cloud layer features from the ceilometer and adopted random forests for classification.
To represent the cloud image more effectively, Li et al.  put forward a discriminative model based on a bag of microstructures (BoMS), and it showed competitive performance in the cloud-type recognition. To make up the weakness of BoMS that it could not describe these complex categories well, the duplex norm-bounded sparse representation model  was reported. This model demonstrated promising classification performance and was validated to be capable of capturing the most prominent patterns from complex cloud categories and naturally to attain a higher accuracy.
Recently, the symmetric positive definite (SPD) matrix manifold has gained much popularity in the action recognition, object detection, face recognition, etc. [20, 21]. In addition, sparse representation on SPD matrix manifolds has been applied to these areas to achieve better performances [22, 23]. In spite of its effectiveness, the matrix manifolds method is seldom investigated to address the task of cloud classification .
In this paper, manifold kernel sparse coding and dictionary learning (MKSCDL) on the SPD matrix manifold is proposed for ground-based cloud classification. The rest of this paper is organized as follows: Section 2 introduces the dataset, and Section 3 describes the methodology of MKSCDL. Section 4 reports the experimental results and discussions. Ultimately, conclusions are summarized in Section 5.
2.1. Zenithal Dataset
The zenithal dataset is provided by the National University of Defense Technology in China and acquired from historical ground-based infrared images taken by the whole-sky infrared cloud-measuring system (WSIRCMS) . The images are grouped into five categories: stratiform, cumuliform, waveform, and cirriform clouds and clear sky according to the morphology and generating mechanism of the cloud . This dataset is composed of 100 cloud images in each category. The typical samples with a size of pixels from each category are demonstrated in Figure 1.
2.2. SWIMCAT Dataset
The SWIMCAT dataset  contains 784 images taken from a daytime WSI called the wide-angle high-resolution sky-imaging system (WAHRSIS). The images are classified into 5 distinct categories: clear sky, patterned clouds, thick dark clouds, thick white clouds, and veil clouds. The number of images in per category is 224, 89, 251, 135, and 85, correspondingly. The images were obtained over a period from January 2013 to May 2014. They were selected based on visual characteristics and were categorized with the help of experts from Singapore Meteorological Services. The representative samples with a size of pixels from each category are demonstrated in Figure 2.
In this section, the methodology is introduced mainly in three parts: feature extraction, dictionary learning, and classification, which is illustrated overall in Figure 3.
3.1. Feature Extraction and Stein Kernel
Given an image with a size of , the feature image is defined by computing d-dimensional features at every pixel:where is the feature mapping, for example:where is the pixel location; represents the pixel gray value; , , , and denote the first- and second-order derivatives of in the direction of and , respectively; and means the modulus of the gradient.
The covariance descriptor (CovD) of the feature image is computed by the following equation:where represents the mean feature vector and is the pixel amount of the image.
In general, the CovD is an SPD matrix. All SPD matrices form a Riemannian manifold when endowed with a Riemannian metric. Note that the SPD matrix is adopted as the extracted feature to describe the image; therefore, it is different from traditional features used for classification in the Euclidean space.
In this paper, we adopt the Stein divergence as a Riemannian metric, and the SPD matrix manifold is mapped into the reproducing kernel Hilbert space (RKHS). The Stein divergence is defined as follows:where and are the points on the SPD matrix manifold, and measures the distance between these two points.
The Stein kernel is defined as follows:
It is a positive definite kernel for certain choices of . With the Stein kernel, we can map the SPD manifold into RKHS:
3.2. Kernel Sparse Coding and Dictionary Learning
In this section, we give a framework for manifold kernel sparse coding and dictionary learning (MKSCDL), which is outlined in Algorithm 1. Let denote points on and be a dictionary with atoms. With the Stein kernel, we update the dictionary by two steps iteratively: kernel sparse coding and kernel dictionary learning. The model is an optimization problem with constraint:where is a sample from , represents the sparse coefficient, is a regularization parameter, and denotes the reconstruction error.
3.2.1. Kernel Sparse Coding
When the dictionary is fixed, the sparse coding problem is formulated as follows:
The iterative shrinkage-thresholding algorithm (ISTA)  is adopted for the optimization solution.
Let , then the sparse vector is updated as follows:where is the step size, represents the sparse coefficient at the -th iteration, and the shrinkage operator is defined as follows:
Equation (10) is equal to .
Now, the problem is transformed to calculate the gradient of with respect to :
As , the gradient of with respect to is
The first term of Equation (12) is
Similarly, the second term of Equation (12) is
3.2.2. Kernel Dictionary Learning
First, the initial dictionary is achieved by -means on Riemannian manifolds using the Frechet mean . It selects k points as the initial cluster centers from the training data randomly. Then measured by Stein divergence, every point is allocated to the closest cluster center to recompute the corresponding cluster center iteratively based on the Frechet mean via the following equation:where is a set of SPD matrices and is the updated cluster center. Ultimately, the forms the initial codebook .
When the sparse coefficient is fixed, the dictionary is updated atom by atom, and the dictionary learning problem is formulated as follows:
We use ISTA  to update the dictionary as well. The Euclidean gradient of is
As proved in , the Riemannian gradient is
The i-th atom is updated as follows:where is the step size and represents the -th atom at the -th iteration.
After the dictionary is learned from the training set, the sparse coefficient and the reconstruction errors “RE” of the testing set are obtained to predict the cloud type. Algorithm 2 details the procedure of classification.
Since there are types of the samples, the dictionary as well has types, which are updated independently, as detailed in Section 3.2.2.
The sparse coefficient of the testing sample is computed as follows:
With the sparse coefficients on each kind of dictionary, the smallest reconstruction error indicates the class that the testing sample belongs to.
In this paper, the number of cloud classes in the two datasets is both 5, and the reconstruction errors on 5 classes of the dictionary are compared to decide the cloud type.
4. Results and Discussions
In this section, compared with the baselines [11, 12, 27], the performance of MKSCDL is evaluated with the same experimental setup on two different image datasets: zenithal dataset, captured by WSIRCMS, and SWIMCAT dataset, gathered by WAHRSIS. Each experiment is implemented three times with 10-fold cross validation, and average values are treated as final results.
Note that the feature map is defined independently on two different datasets due to the different nature between the grayscale and colour images. is defined in Equation (2) to the grayscale image in the zenithal dataset, while in the SWIMCAT dataset to the RGB image, where , , and represent the intensity images in three channels, respectively. The parameters used in our experiments are set empirically as follows: , , , and .
4.1. Results of the Zenithal Dataset
4.1.1. Performance of MKSCDL
The first experiment is carried out on the zenithal dataset. With diverse choices of the atom number in the dictionary, the performance of MKSCDL varies correspondingly. Figure 4 shows the overall accuracy on the updated dictionary with different sizes. It is illustrated that when the dictionary size equals 14, the overall accuracy achieves up to 96.33%, which outperforms the other cases.
Figure 5 reports the confusion matrix when . The element of row i and column j in the confusion matrix means the percentage of the i-th class recognized as the j-th class. As a result, the diagonal elements correspond to the recognition rates of all categories. It is illustrated all of the stratiform clouds and 99.3% clear sky images are identified correctly, which means that these two cloud types possess prominent characteristics to be distinguished. Likewise, the recognition rates of cumuliform, waveform, and cirriform clouds achieve more than 93%. On the whole, MKSCDL reveals an ideal performance in identifying the ground-based grayscale cloud images with a rather high accuracy.
4.1.2. Performance Comparison with the Baselines
To assess the effectiveness of MKSCDL further, WLBP  and BC  are applied for comparison:(i)WLBP  is the method fusing the variance of a local patch into LBP. The KNN classifier is employed for cloud classification based on the chi-square distance.(ii)BC  integrates statistical and local texture features and adopts the Bayesian classifier with regularized discriminant analysis. Note that the statistical features have only 8 dimensions because the infrared images are lack of colour information.
Table 1 presents the comparison results of 10-fold cross validation. The performance of MKSCDL exceeds that of the other two methods especially with regard to cumuliform, waveform, and cirriform clouds. MKSCDL achieves the highest overall accuracy of 96.33% among them. That means the dictionary is learned well and the samples are described adequately on the corresponding dictionary instead of different types of dictionary, which makes contribution to the competitive performance.
4.2. Results of the SWIMCAT Dataset
4.2.1. Performance of MKSCDL
The second experiment is conducted on the SWIMCAT dataset. Similar to the first experiment, Figure 6 exhibits the overall accuracy on the learned dictionary with different sizes. As the dictionary size increases, the overall accuracy improves in general. When is 20, MKSCDL performs best with a result of 98.34%. In consideration of excellent classification performance and computational cost of MKSCDL, we confirm that can satisfy the experimental requirement.
Figure 7 demonstrates the confusion matrix when . Patterned clouds and thick white clouds possess obvious characteristics for discrimination with an accuracy of 100%. Likewise, the results of the clear sky and thick dark clouds achieve over 98%. In addition, the challenging veil clouds resembling clear sky  attain a decent result of 92.94%. It is shown that the misclassification rate for each class is rather low, which means that on the whole, the proposed method works well in categorizing the ground-based RGB images.
4.2.2. Performance Comparison with the Baselines
Besides the two baselines mentioned in Section 4.1.2, the texton-based method  integrating both colour and texture features is used for comparison as well. Note that different from the grayscale images, the statistical features extracted from colour images in the BC method have 12 dimensions.
Table 2 presents the comparison results of 10-fold cross validation. By contrast, MKSCDL performs better than WLBP and BC overall. It is clear that MKSCDL has a strong power for the task of cloud categorization of the SWIMCAT dataset.
To compare with the texton-based approach well, we also implement the experiment with the same configuration as that in , which chose a training set with 40 images per category while another 45 ones for testing randomly (40/45). Table 3 lists the results. In comparison with Table 2, the overall performance of different methods changes little. The texton-based method achieves perfect classification accuracy for all categories except veil clouds, whose accuracy remains to be improved. Viewing the results of MKSCDL, each cloud type acquires a fair high recognition rate.
Comprehensively, in comparison with the other three methods, MKSCDL is validated to be the most effective one in recognizing the ground-based colour images.
In this paper, a novel cloud classification named “MKSCDL” on manifolds is proposed. The SPD matrix is extracted from each image and acts as the image feature. To maintain the non-Euclidean geometrical characteristics of the SPD matrix, kernel sparse coding and dictionary learning are conducted to obtain a representative dictionary. The testing sample’s reconstruction errors on different categories of dictionary are calculated to identify the specific cloud type. By comparing recent methods on grayscale and colour datasets, it is interesting to find that WLBP performs better in grayscale images, while BC performs better in colour images. Comparatively speaking, MKSCDL is suitable for both grayscale and colour images and is equipped with high capacity for automatic ground-based cloud classification.
The proposed method MKSCDL can be applied to the real life, and the results provided by the visual observation are adopted for comparison and evaluating the automatic classification method. With limited images, it may not produce perfect performance in cloud-type recognition. As the dataset becomes more representative and more adequate, it will work better and can satisfy the task of cloud classification well. In the future, multiple improvements should be considered for better automatic cloud-type classification. As for the feature extraction, more features like the gray-level cooccurrence matrix and Gabor filtering could be fused into the feature mapping to describe the image well. In terms of dictionary learning, the interclass difference can be taken into consideration for a more discriminative dictionary. What is more, samples’ sparse coefficients can be applied to build the SVM model. In addition, complex sky condition existing in multiple cloud categories is supposed to attract our attention in the next work.
The SWIMCAT dataset is linked to http://vintage.winklerbros.net/swimcat.html, and the zenithal dataset used to support the findings of this study is available from the first author (email@example.com) upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
The authors would like to thank Prof. Yee Hui Lee for providing the SWIMCAT dataset and Dr. Lei Liu for polishing the manuscript. This work was supported by the National Natural Science Foundation of China (Grant nos. 61473310, 41174164, and 41575024).
- G. L. Stephens, “Cloud feedbacks in the climate system: a critical review,” Journal of Climate, vol. 18, no. 2, pp. 237–273, 2005.
- A. J. Teuling, C. M. Taylor, J. F. Meirink et al., “Observational evidence for cloud cover enhancement over western European forests,” Nature Communications, vol. 8, Article ID 14065, 2017.
- B. A. Baum, P. F. Soulen, K. I. Strabala et al., “Remote sensing of cloud properties using MODIS airborne simulator imagery during SUCCESS: 2. cloud thermodynamic phase,” Journal of Geophysical Research Atmospheres, vol. 105, no. 9, pp. 11767–11780, 2000.
- J. T. Houghton, Y. Ding, D. J. Griggs et al., Climate Change 2001: The Scientific Basis, Cambridge University Press, Cambridge, UK, 2001.
- J. Calbó and J. Sabburg, “Feature extraction from whole-sky ground-based images for cloud-type recognition,” Journal of Atmospheric & Oceanic Technology, vol. 25, no. 1, pp. 3–14, 2008.
- M. Xia, W. Lu, J. Yang, Y. Ma, W. Yao, and Z. Zheng, “A hybrid method based on extreme learning machine and k-nearest neighbor for cloud classification of ground-based visible cloud image,” Neurocomputing, vol. 160, pp. 238–249, 2015.
- K. A. J. Buch, C. H. Sun, and L. R. Thorne, “Cloud classification using whole-sky imager data,” in Proceedings of the Fifth Atmospheric Radiation Measurement (ARM), Science Team Meeting, San Diego, CA, USA, 1995.
- M. Singh and M. Glennen, “Automated ground-based cloud recognition,” Pattern Analysis and Applications, vol. 8, no. 3, pp. 258–271, 2005.
- A. Heinle, A. Macke, and A. Srivastav, “Automatic cloud classification of whole sky images,” Atmospheric Measurement Techniques, vol. 3, no. 3, pp. 557–567, 2010.
- R. Liu and W. Yang, “A novel method using gabor-based multiple feature and ensemble SVMs for ground-based cloud classification,” in Proceedings of Seventh International Symposium on Multispectral Image Processing and Pattern Recognition (MIPPR2011), Guilin, China, December 2011.
- S. Liu, Z. Zhang, and X. Mei, “Ground-based cloud classification using weighted local binary patterns,” Journal of Applied Remote Sensing, vol. 9, no. 1, Article ID 095062, 2015.
- H. Y. Cheng and C. C. Yu, “Block-based cloud classification with statistical features and distribution of local texture features,” Atmospheric Measurement Techniques, vol. 8, no. 3, pp. 1173–1182, 2015.
- L. Liu, X. Sun, F. Chen, S. Zhao, and T. Gao, “Cloud classification based on structure features of infrared images,” Journal of Atmospheric & Oceanic Technology, vol. 28, no. 3, pp. 410–417, 2011.
- W. Zhuo, Z. Cao, and Y. Xiao, “Cloud classification of ground-based images using texture-structure features,” Journal of Atmospheric & Oceanic Technology, vol. 31, no. 1, pp. 79–92, 2014.
- Y. Xiao, Z. Cao, W. Zhuo, L. Ye, and L. Zhu, “mCLOUD: a multiview visual feature extraction mechanism for ground-based cloud image categorization,” Journal of Atmospheric & Oceanic Technology, vol. 33, no. 4, pp. 789–801, 2016.
- A. Kazantzidis, P. Tzoumanikas, A. F. Bais, S. Fotopoulos, and G. Economou, “Cloud detection and classification with the use of whole-sky ground-based images,” Atmospheric Research, vol. 113, no. 1, pp. 80–88, 2012.
- J. H. Tato, F. J. R. Benítez, C. A. Barrena, R. A. Mur, I. G. Leon, and D. P. Vázquez, “Automatic cloud‐type classification based on the combined use of a sky camera and a ceilometer,” Journal of Geophysical Research: Atmospheres, vol. 122, no. 20, pp. 11045–11061, 2017.
- Q. Li, Z. Zhang, W. Lu, J. Yang, Y. Ma, and W. Yao, “From pixels to patches: a cloud classification method based on a bag of micro-structures,” Atmospheric Measurement Techniques, vol. 9, no. 2, pp. 753–764, 2016.
- J. Gan, W. Lu, Q. Li et al., “Cloud type classification of total-sky images using duplex norm-bounded sparse coding,” IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, vol. 10, no. 7, pp. 3360–3372, 2017.
- S. Jayasumana, R. Hartley, M. Salzmann, H. Li, and M. T. Harandi, “Kernel methods on riemannian manifolds with gaussian RBF kernels,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 37, no. 12, pp. 2464–2477, 2015.
- M. Faraki, M. Palhang, and C. Sanderson, “Log-Euclidean bag of words for human action recognition,” IET Computer Vision, vol. 9, no. 3, pp. 331–339, 2014.
- A. Cherian and S. Sra, “Riemannian dictionary learning and sparse coding for positive definite matrices,” IEEE Transactions on Neural Networks & Learning Systems, vol. 28, no. 12, pp. 2859–2871, 2017.
- Y. Wu, Y. Jia, P. Li, J. Zhang, and J. Yuan, “Manifold kernel sparse representation of symmetric positive-definite matrices and its applications,” IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, vol. 24, no. 11, pp. 3729–3741, 2015.
- Q. Luo, Y. Meng, L. Liu, X. Zhao, and Z. Zhou, “Cloud classification of ground-based infrared images combining manifold and texture features,” Atmospheric Measurement Techniques Discussions, vol. 11, pp. 5351–5361, 2018.
- X. J. Sun, T. C. Gao, D. L. Zhai, S. J. Zhao, and J. G. Lian, “Whole sky infrared cloud measuring system based on the uncooled infrared focal plane array,” Infrared & Laser Engineering, vol. 37, no. 5, pp. 760–764, 2008.
- X. Sun, L. Liu, T. Gao, and S. Zhao, “Classification of whole sky infrared cloud image based on the LBP operator,” Transactions of Atmospheric Sciences, vol. 32, no. 4, pp. 490–497, 2009.
- S. Dev, Y. H. Lee, and S. Winkler, “Categorization of cloud image patches using an improved texton-based approach,” in Proceedings of IEEE International Conference on Image Processing (ICIP), Quebec City, Canada, September 2015.
- S. Zhang, S. Kasiviswanathan, P. C. Yuen, and M. T. Harandi, “Online dictionary learning on symmetric positive definite manifolds with vision applications,” in Proceedings of Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 3165–3173, AAAI Press, Austin, TX, USA, January 2015.
- M. Faraki, M. T. Harandi, and F. Porikli, “More about VLAD: a leap from euclidean to riemannian manifolds,” in Proceedings of Computer Vision and Pattern Recognition (CVPR), Boston, USA, January 2015.
- A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM Journal on Imaging Sciences, vol. 2, no. 1, pp. 183–202, 2009.
Copyright © 2018 Qixiang Luo et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.