Abstract

Two-dimensional principal component analysis algorithm (2DPCA) can be performed in the batch mode and can not meet the real-time requirements of the video stream. To overcome these limitations, the incremental learning of the candid covariance-free incremental PCA (CCIPCA) is innovated to the existing 2DPCA, and the called incremental 2DPCA (I2DPCA) is firstly presented to incrementally compute the principal components of a sequence of samples directly on the 2D image matrices without estimating the covariance matrices. Therefore, the I2DPCA can improve the feature extraction speed and reduce the required memory. However, the variations between the column direction, generally neglected, are also useful for the high-accuracy object recognition. Thus, another incremental sequential row-column 2DPCA algorithm (IRC2DPCA), based on the proposed I2DPCA algorithm, is also proposed. The IRC2DPCA can compress the image matrices in the row and column direction, and the feature matrices extracted by the IRC2DPCA are with less dimensions than the I2DPCA. The substantial experimental results show that the IRC2DPCA, compared with the other three algorithms, can improve the convergence rates and the recognition rates, compress the dimensions of the feature matrices, and reduce the feature extraction time and the classification time.

1. Introduction

Object recognition has been an active research field in computer vision to classify the objects from the images or the sequent arriving video frames, widely used in the pattern recognition fields of human face [1, 2] and authentication [35]. Since the dimension of the original image space obtained by the camera is high, the dimension-reduction and feature extraction algorithms are usually employed before classification and recognition to improve the object recognition speed. It can be widely applied in the fault diagnosis scheme for planetary gearboxes [6], the early fault diagnosis of rolling bearings [7], and the thermal nondestructive testing imagery [8].

Principal component analysis (PCA) is a classical feature extraction and dimension-reduction algorithm. Usually, the eigenvectors corresponding to the larger eigenvalues are selected as the projection axes, and the high dimension data can be projected to the lower dimension subspaces for reducing the dimensions. The PCA, combined with the infrared thermograph, has been used in various application cases [911]. For the PCA-based algorithms, the 2D image matrices must be transformed into the 1D column vectors to construct the covariance matrices. Concatenating the entries in the 2D matrices into the 1D vectors often leads to the high-dimensional sample spaces, and it is difficult to accurately evaluate the eigenvectors of the covariance matrices. Furthermore, computing directly the eigenvectors of the covariance matrix is very time-consuming [12]. Yang et al. [12] proposed a two-dimensional principal component analysis algorithm (2DPCA), which can extract the eigenvectors directly from the image matrices without the image-to-vector transformations. It can preserve the spatial structure feature of the images, reduce the dimensions of the covariance matrix, and solve the small sample size problem of the PCA [13, 14]. Subsequently, several 2DPCA extended algorithms were proposed [1517]. All these extended algorithms ignored the longitudinal feature of the images, and the image information, contained in the feature matrices, is not comprehensive enough. To overcome the weaknesses of the 2DPCA, Yang et al. [18] proposed the sequential row-column 2DPCA algorithm (RC2DPCA). It can completely preserve the feature information of the images by the twofold feature extractions in the row and column directions and can obtain the feature matrices with less dimensions.

In terms of the aforementioned PCA, 2DPCA and RC2DPCA, these algorithms are performed in batch modes. It means that all training sample data have to be available before the principal components can be estimated. If we incorporate additional training data into the existing projection matrix, the matrix has to be retrained with the whole training data. For these batch algorithms, the extraction time and the required memory will increase rapidly with the increase of the training data. Eventually, it will lead to the problem that the feature extraction speed can not meet the updating speed of the training data [19]. The incremental PCA (IPCA) does not need to obtain all training data before the feature extraction; the feature information is updated gradually with the sequent input of the samples. Compared with the batch algorithms, the IPCA can reduce the required memory because it does not need to save the historical training data. In addition, the IPCA can make full use of the historical training results for the subsequent training, and the appended training time can be greatly reduced. The first category of the IPCA algorithms [2023] allows for the incremental learning using the eigenspace method. It can update the coefficients of the images using the updated eigenspaces without having to retain the original images. Since the new samples are added one by one and the least significant principal components are discarded to preserve the dimensions of the subspaces, the method also suffers from the problem of unpredicted approximation error. The second category can estimate the principal components without directly computing the covariance matrix [24, 25]. The candid covariance-free IPCA (CCIPCA) algorithm developed by Weng et al. [24] can estimate the principal components of the high-dimensional image matrices quickly with a good convergence performance. The candid covariance-free incremental principal component thermography (CCIPCT) [8, 26] presented the application of the CCIPCA in the thermography, and it can provide lower computational complexity as compared to the principal component thermography (PCT). Li et al. [27] proposed an extension method based on the CCIPCA to incrementally update the feature extractors for the camera identification. However, there exist the problems of high-dimensional image vectors since the eigenvectors are statistically determined by the covariance matrix, no matter which method is adopted for obtaining the matrix.

In this paper, an incremental 2DPCA algorithm (I2DPCA) is developed. It can overcome the limitations of the IPCA and the batch 2DPCA. The proposed I2DPCA can compute incrementally the principal components directly on the 2D image matrix without estimating the covariance matrix. It can reduce the extraction time and the required memory. However, the I2DPCA only compresses the image matrices in the row direction, and it requires more coefficients for the image representation. It will lead to a slow classification speed and a large memory requirement for the large-scale databases. Therefore, another incremental RC2DPCA algorithm (IRC2DPCA), based on the I2DPCA algorithm, is proposed to reduce the dimension of the feature matrices by the twofold feature extraction in the row and the column directions. Both the extraction time and classification time are reduced by the IRC2DPCA. Finally, using the block database and the ORL face database as the experimental samples, the convergence rates of the principal components, the classification rates, the computation time, and the required memory are compared and analyzed.

2. The Proposed I2DPCA Algorithm

2.1. The 2DPCA Algorithm

The 2DPCA, proposed by Yang et al. [12], can directly compute the covariance matrix for the image matrices without the image-to-vector transformation. Consider the training set which consists of training image matrices , and the size is . The covariance matrix in the row direction can be constructed directly bywhere represents the mean of all the training image matrices. Note that is a matrix. Since is a positive semidefinite matrix, there exists an orthogonal matrix which satisfies the equation . The eigenvectors corresponding to the first largest eigenvalues of are selected to construct the projection matrix ,

Projecting onto , we can obtain a feature matrix as

The 2DPCA can compute directly the covariance matrix without transforming the image matrices into vectors. However, the 2DPCA is a batch algorithm, therefore all the image sample matrices should be acquired in advance and saved before training; the existing covariance matrix has to be retrained with all training data when a new image is appended. With the increase of the training data, the exaction speed can not meet the updating speed of the new training data. Hence, it can not be applied to the training of the continuous high-dimensional sample data or the real-time video stream data.

2.2. The Proposed I2DPCA Algorithm

To overcome the limitations of the batch learning and the real-time problem, we propose a I2DPCA algorithm based on the CCIPCA [24]. Suppose that the 2D training image matrices are acquired sequentially, , are possibly infinite, and each 2D image matrix , , is of size . Assume that the input samples satisfy the zero mean Gaussian distribution [24], and these samples must be centralized in advance. The centralization matrices can be expressed by . The covariance matrix in the row direction from the samples can be constructed aswhere is . Different from the 2DPCA, it can only be estimated as an intermediate step without directly computing. The eigenvector of the matrix satisfieswhere is the corresponding eigenvalue. By replacing the of (4) and replacing the of (5) with its estimator at each step , we can rewrite (5) aswhere is the step estimator of . Considering , and , we can approximately estimate as . The following incremental form can be given as

For the incremental estimation, (7) can be written as the following recursive form:where is a positive parameter called the amnesic parameter. When a new image matrix is appended, the mean matrix needs to be updated. The incremental recursive form of the mean matrix can be updated byEquation (8) only can estimate the first dominant eigenvector. To compute the eigenvector, the 2D image matrix is subtracted from its projection on the order estimated eigenvector aswhere . The obtained residual is used as the input data of the iteration step. In this way, the first dominant eigenvectors are estimated, and the projection matrix can be obtained by

Projecting onto , we can obtain the feature matrix as

The I2DPCA does not need to obtain all the sample data at once before training. Instead, a recursive update of the eigenvector can be performed when a new image sample is input. Therefore, the algorithm does not save all the image samples, and it can reduce the required memory. Without directly computing the covariance matrix, the I2DPCA can estimate the eigenvectors of the training image matrices in a recursive form. However, the I2DPCA only works in the row direction of the image matrices, ignoring the information in the column direction. Furthermore, the dimension of the feature matrix, extracted by the I2DPCA, is still high ( is large). More time and more memory are needed in the classification process.

3. The Proposed IRC2DPCA Algorithm

The I2DPCA can realize the incremental learning and reduce the feature extraction time and the required memory. However, the classification time and the required memory are still large due to the high dimension of the feature matrix extracted by the I2DPCA. And the I2DPCA only considers the transverse features of the image sample matrix but neglects its longitudinal features, so that the extracted image feature information is not comprehensive enough. Therefore, based on the I2DPCA, the feature matrix which has been extracted in the row direction continues to be extracted in the column direction so that the final feature matrix contains enough information of the images in both the row and column directions.

The feature matrices , , extracted by the I2DPCA, are used as the training matrices of the IRC2DPCA. express the centralization matrices. And then the covariance matrix , different from of the I2DPCA, can be expressed as

The incremental form with the amnesic parameter can be expressed by

Similarly, the updating mean matrix can be obtained by the following recursive form:

We can compute the high order eigenvectors associated with matrix by generating observations only in the complementary space. In this way, the first dominant eigenvectors are estimated, and the projection matrix in the column direction can be constructed by

The feature matrix, extracted by the twofold feature extractions in the sequential row and column directions, can be computed by

The resulting feature matrix is a matrix, much smaller than the I2DPCA feature matrix . The IRC2DPCA algorithm can be described as follows.

Step 1 (feature extraction in the row direction). Using formula (8) of the I2DPCA, the first dominant estimators can be obtained sequentially with the sequent input of the original image matrix . The first higher order estimators can also be computed by formula (10) with the increase of the number of the order. The first eigenvectors corresponding to the first samples can be gotten by to construct the matrix . And then the feature matrix can be obtained by formula (12).

Step 2 (feature extraction in the column direction). Using the results of the I2DPCA as training matrix , the IRC2DPCA can compute the first dominant estimators by the formula (14). Similarly, we can get the second projection matrix by the formula (16).

Step 3 (the final feature matrix). Project all the original image matrices onto the matrices of and , and the final feature matrices can be gotten by formula (17).

The incremental learning of the IRC2DPCA can satisfy the learning of the continuous high-dimensional sample data or the real-time video stream data. At the same time, the algorithm allows the information of the image matrices in the row and column directions to intermingle without losing their separate identities. In addition, the dimension of the feature matrices is , much smaller than that of the I2DPCA. It can effectively reduce the dimension of the feature matrices and the classification time.

4. Experiments and Analysis

Two sets of databases are used to evaluate the performances of the proposed algorithms. The block database contains four kinds of blocks (the green cylinder, the green triangular prism, the red cube, and the yellow cuboid), each having 500 different samples. The size of each image is pixels, and all images are grayscale processed before the experiments. Figure 1 shows the 10 images of each kind of blocks. The ORL face database contains images from 40 persons, each providing 10 different images. The size of each image is pixels. Figure 2 shows the 40 images in the ORL. All programs run on the computer whose CPU is E5-2620 v4, the main frequency is 2.10GHz, and the memory is 32 GB. The operation system is 64-bit Windows 10, and the programming environment is VS2010+Opencv2.4.10.

4.1. Experiments on the Block Database

The 2DPCA [12] and the RC2DPCA [18] are only batch algorithms without the convergence. The amnesic parameter is set to be 2, and the convergence experiments are performed on the I2DPCA and the IRC2DPCA, respectively. The correlation between the estimated eigenvectors and the accurate ones , computed by the batch methods is represented by their inner product or . The larger the inner product, the more correlated the estimated and the accurate eigenvectors.

In the experiment, 500 red cube samples are input in turn. The convergence rates of the I2DPCA and the IRC2DPCA are shown in Figures 3 and 4, respectively. It can be seen that the first five eigenvectors of the IRC2DPCA show an obvious convergent tendency, while the later three ones have poor convergence. The convergence rates of the first five eigenvectors of the IRC2DPCA can reach 90.6% after all samples are input incrementally. For the I2DPCA, only the first three eigenvectors have an obvious convergence. The convergence rates of the five ones can only reach 77.7%. Hence, the estimated eigenvectors of the IRC2DPCA are closer to the accurate ones than those of the I2DPCA.

The convergence rate is influenced by the sample-to-dimension ratio [24]. The higher the ratio, generally, the easier the convergence. In this experiment, the original sample image is pixels. The image dimension of the 2DPCA is , much higher than that of the IRC2DPCA (). Thus, when the sample number reaches 500, the sample-to-dimension ratio of the IRC2DPCA is 0.521, while that of the I2DPCA is only 0.035. With the same number of samples, the sample-to-dimension ratio of the IRC2DPCA is larger than the I2DPCA, and the convergence rate of the IRC2DPCA is higher than that of the I2DPCA.

To improve the convergence of the algorithms, 500 red cube samples are input repeatedly; the convergence rates of the I2DPCA and the IRC2DPCA are shown in Figures 5 and 6, respectively. We can see that the first eight eigenvectors of the IRC2DPCA show a more obvious tendency to converge than those of the I2DPCA. To achieve a higher convergence rate, the sample number should be at least 1000. If the video stream is input, the video should last for at least 37s.

The classification experiments are performed on the block database and the k-nearest neighbour algorithm (KNN) is employed for classification. Through many experiments, we can find that the feature matrices dimension always fluctuates around a certain value and the fluctuation value is very small when the highest classification rates can be obtained for the four algorithms. Therefore, in the experiment, a specific feature dimension for each algorithm is selected for testing. L samples are randomly selected from each kind of blocks for training, and the remaining ones are used for testing. To ensure the adequate training samples, is needed.

The highest classification rates and the corresponding feature matrix dimensions are given in Table 1. The classification rates of the four algorithms can remain on an upward tendency with the increase of the samples. The performance of the IRC2DPCA is superior to the other three algorithms in the classification rate. When the highest classification rate of 99.8% is obtained, the feature matrix dimension of the IRC2DPCA is only , far smaller than that of the I2DPCA; that is, .

The IRC2DPCA is also superior to the I2DPCA in the classification time. Table 2 shows that the average classification time per image of the I2DPCA is 0.0015s, and the IRC2DPCA has less average classification time of 0.0002s, mainly because the IRC2DPCA has a feature matrix with relatively fewer dimensions. However, the average feature extraction time per image of the IRC2DPCA is 0.0516s, slower than that of the I2DPCA which is 0.0267s, because the IRC2DPCA needs to do the feature extraction in both the row and column directions.

Table 3 shows the required memory in the feature extraction and the classification. It can be seen that the memory of the 2DPCA and the RC2DPCA will increase with the increase of the samples, and this is because both algorithms are batch algorithms. The required memory of the I2DPCA and IRC2DPCA remains basically unchanged. When facing large samples, the advantages of the two incremental algorithms in memory will be more significant.

4.2. Experiments on the ORL Database

To verify the generality of the proposed algorithms, a series of experiments are performed on the ORL face database. In the experiments, 400 face images are input ten times. The convergence rates of the I2DPCA and the IRC2DPCA can be shown in Figures 7 and 8, respectively. Compared with the I2DPCA, the IRC2DPCA has better convergence performance on the ORL database, similarly.

To better compare the performance of the I2DPCA and the IRC2DPCA, we use the two algorithms to reconstruct the face images. By adding up the first subimages, we can obtain the approximate reconstruction images of the original images. Figure 9 shows eight reconstructed images of the first face image shown in Figure 2. The reconstruction images become clearer with the increase of the subimages. Because the feature extraction dimension of the IRC2DPCA is much smaller than that of the I2DPCA, the effect of the IRC2DPCA on the image reconstruction is poorer.

From the classification rates shown in Table 4, the IRC2DPCA has better performance in the classification rate. When the highest classification rate reaches 97.5%, the feature matrix dimension of the IRC2DPCA is far smaller than that of the I2DPCA. For the running time, Table 5 shows that the IRC2DPCA has less average classification time of 0.0002s than the I2DPCA of 0.0015s. The classification time of the IRC2DPCA also has a great advantage on the ORL database. As shown in Table 6, we can see that, in the running memory, the advantages of the two incremental algorithms are also obvious with the increase of the samples.

5. Conclusions

In this paper, a new feature extraction algorithm called the I2DPCA for image representation and object recognition has been developed. The I2DPCA can estimate the dominant eigenvectors from incrementally arriving 2D image data without transforming the 2D image matrix into a high-dimensional vector and without directly computing the covariance matrix. It can reduce the complexity of the algorithm and improve the feature extraction time. And then another incremental algorithm called the IRC2DPCA is proposed to intermingle effectively the information in both the row and column directions with less dimensions. The algorithm can effectively reduce the dimension of the feature matrices and the classification time.

The experimental results show that (1) with the increase of the training data, the advantage of the incremental algorithms in the feature extraction time and the required memory is gradually obvious; (2) compared with the I2DPCA, the IRC2DPCA has higher convergence rate and faster convergence speed; (3) the IRC2DPCA has higher classification rate with the less-dimension feature matrix, and its highest classification rate can reach 99.8%; (4) although the IRC2DPCA can not be performed as well in the reconstruction image because the feature matrix dimension is far smaller than the I2DPCA, the classification time is better than the I2DPCA. The proposed algorithms have a widespread prospect in the equipment fault identification and the robot vision tracking. Applying the algorithms, the recognition of the industrial weld surface defect is being tested by other members of our project. The IRC2DPCA algorithm can provide an effective solution for the real-time learning of the high-dimensional sample data or the video stream data.

Nomenclature

PCA:Principal component analysis
2DPCA:Two-dimensional principal component analysis
RC2DPCA:Sequential row-column 2DPCA
IPCA:Incremental PCA
CCIPCA:Candid covariance-free IPCA
CCIPCT:Candid covariance-free incremental principal component thermography
PCT:Principal component thermography
I2DPCA:Incremental 2DPCA
IRC2DPCA:Incremental RC2DPCA
KNN:k-nearest neighbour.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This work is supported by National Key R & D Plan of China (Grant No. 2017YFB1303304), Tianjin Science and Technology Planed Project of China (Grant No. 17ZXZNGX00110), and Tianjin Natural Science Foundation Key Project of China (Grant No. 16JCZDJC30400).