Abstract
The agriculture field is one of the most important fields where computational techniques play an imperative role for decision-making whether it is the automation of watering of plants, controlling of humidity levels, and detection of plant diseases and growth of plants. There are problems in the conventional methods where newer computational techniques and image processing methods are not used to keep track of growth of plants. The traditional image capturing and processing models have problems of large image segmentation error, excessive feature extraction time, and poor recognition output. In order to overcome the problems in the traditional plant growth methods based on image processing automations, computer image processing with computational method has been proposed to analyze the plant growth by utilizing state recognition method for corn and rice crops. An image acquisition platform is established on the basis of CMOS image sensor for crop image acquisition. The binary processing is performed, and then the images are segmented to reduce error of segmentation results in the traditional methods. To extract image features of corn and rice crops, convolution neural network (CNN) with newer architecture is used. According to contour information of images, the block wavelet transform method is used for feature adaptive matching. The binary tree structure is used to divide the growth period of corn and rice crops. The fuzzy mathematical model is also devised to identify the characteristics of crops in different growth periods and to complete the identification of growth state. Experimental results show that the proposed method effectively improves problems of traditional methods with better image recognition effect and reduces the time of feature recognition. The time to extract features by the proposed method is 1.4 seconds, whereas comparative methods such as random forest (RF) take 3.8 s and other traditional techniques take 4.9 s. Segmentation result error of the recognition method is also reduced significantly.
1. Introduction
Automatic identification of crop growth period is one of core parts of precision agriculture support technology [1]. Traditional crop growth period identification is mainly recorded through manual observation, which has problems such as time-consuming and laborious, low efficiency, strong human subjectivity, different observation standards, and difficult to ensure measurement accuracy [2]. With mature application of sensor detection technology and remote network transmission technology, crop observation is gradually transitioning from manual observation to automatic observation. However, there are still problems of low efficiency, observation accuracy, and observation frequency [3]. At present, computer vision technology is mainly used for the classification and recognition of crop growth period [4]. Because taking crop images in the field requires fixed shooting equipment and shooting at the same distance, it has high requirements for light and shooting angle, and the recognition effect is poor [5]. These problems are observed by many researchers and relevant scholars have conducted a lot of research.
The authors propose a crop hyperspectral remote sensing recognition method based on the random forest method [6]. The random forest method is used to analyze the reflection spectra of 8 typical crops, extract and classify the characteristic bands, and compare the recognition effects of different methods. Results show that the random forest method does not need to preprocess the reflection spectrum but directly processes the full band reflection spectrum data. It not only selects the characteristic bands to distinguish different crops but also uses the selected bands to classify crops. While showing the advantages of hyperspectral remote sensing in identifying crops, it also provides reference for remote sensing fine classification of large-area crops. However, due to the influence of illumination, this method has the problem of poor recognition effect in the recognition process, and some details of the image are blurred. Researchers have proposed a fast wheat identification method based on GF-2 data, using GF-24m multispectral remote sensing images as data source using supervised classification methods [7] (including support vector machines, artificial neural networks, and maximum likelihood) for rapid extraction and precision analysis of spatial distribution information of wheat planting. The results show that the recognition results of this method have high accuracy and can provide relevant data for the study of crop growth characteristics, but there is a problem of high segmentation errors in crop growth state images.
In [8], authors have proposed a crop canopy recognition model based on thermal infrared image processing technology. Firstly, using adaptive characteristics of the five-layer linear normalized fuzzy neural network, Gaussian membership function is selected to automatically calculate reasoning rules of canopy visible light image recognition, and effectively segment the canopy region in visible light images. Three segmentation indexes and entropy were analyzed to quantitatively evaluate the canopy segmentation quality of visible images. Taking canopy effective area of the obtained visible images as the reference image, the affine transformation algorithm is used to adjust the optimal image transformation factors such as translation, rotation, and scaling and register the original thermal infrared image. A canopy thermal infrared image recognition method based on affine transformation is proposed. Finally, the mutual information of entropy is used as the supervision index to evaluate the recognition method of crop canopy thermal infrared image. Results show that this method can reflect physiological and ecological information characteristics of crops through thermal infrared images and has certain effective practicability. However, there is a problem of long time for crop feature extraction which affects real-time acquisition of recognition results.
Under the background of automatic crop observation requirements, this paper takes corn and rice as research subjects and uses image processing technology to effectively identify the growth state of corn and rice. In the rapidly developing field of computer vision, image processing technology is widely used in various target recognition occasions. Morphological features of the target are extracted by means of image binarization and segmentation, and then feature representation is carried out. Finally, the target recognition is completed, such that it can be well applied in the area of research. The results obtained by the proposed technology are promising. The next section describes the proposed work, followed by the results obtained from the proposed work. Finally, this paper summarizes our research work.
2. Image Preprocessing of the Growing States of Corn and Rice Crops
Image processing technology is very helpful for extracting important or meaningful features of the image. This technology takes any image as input and gives output in terms of its number of features or specifications as per user’s requirement.
2.1. Crop Image Collection
Before identifying the growth state of corn and rice crops, the target image needs to be collected. This paper mainly uses the image collection platform established on the basis of the CMOS image sensor [9] to carry out this operation. The structure of the CMOS image sensor is shown in Figure 1.
The CMOS image sensor is a typical solid-state imaging sensor, which is an important component to realize image acquisition. It is connected with the embedded platform through CMOS interface and controlled by the embedded platform to obtain crop images [10]. The function of the embedded platform is to collect crop images regularly, then fuse, quality judge, and compress the images, and then transmit the processed image data to the data center through 4G network after receiving the image data acquisition instruction from the crop image information acquisition management system. The crop image acquisition platform based on the CMOS image sensor is responsible for image acquisition node management, image data acquisition instruction issuing and receiving data, and image processing. It is also providing users with services such as image retrieval, crop growth analysis, and pest and disease analysis [11]. With reference to crop image processing technology, there is a need to have identifiable difference among the images. For that purpose, growth state identification technique is used. Users can access the crop image acquisition platform through the client (desktop computer, mobile phone, PDA, etc.) to complete the management of image acquisition nodes, image retrieval, crop growth analysis, disease and insect pest analysis, weed analysis, and other applications.
2.2. Crop Image Preprocessing
Image preprocessing is a kind of preprocessing relative to image recognition which uses a series of specific operations to “transform” images according to specific goals. No matter what kind of device is used, the collected images are often unsatisfactory. The captured images may be too blurry, the outline of the object may be too sharp, and the image may be distorted or deformed. The specific target can make the image clearer and can also obtain some specific information from the image. In this study, image preprocessing process is shown in Figure 2.
According to Figure 2, image binarization is the key link of image processing. In order to achieve accurate image segmentation, image binarization processing is required [12]. After grayscale conversion, image input to the computer is a grayscale image. To extract the shape characteristics of crops in images, it is often converted into a binary image, and the target information is obtained from the binary image. Compared with grayscale images, binary images have greatly reduced information, faster image preprocessing, lower cost, and higher practical value [13]. The key to image binarization is to correctly select the threshold. Process of image processing is to take all pixels whose gray value falls within the threshold range, representing the object, and taking the value of other remaining pixels for representing the background. Through binarization process, crops are extracted out from complex image background.
The transformation function expression for image binarization is shown in the following equation:
In the above formula, represents the threshold value during binarization processing. The key to image binarization is the selection of threshold. By selecting an appropriate threshold for image binarization, the features of objects in the image can be highlighted, which facilitates the extraction of feature parameters and at the same time retains as much useful information as possible.
2.3. Crop Image Segmentation
Image segmentation is the key to image processing and also the bottleneck restricting the development of computer image processing technology. Therefore, based on the results of image binarization, the crop images are segmented [14]. The original crop image is represented by a two-dimensional grid with dimensional vectors, where each grid point represents a pixel. When , it is a grayscale image; when , it is a color image; when , it is a multispectral image. The space where the grid is located is called the air domain and the space where the grayscale or spectral information is located is called the chromaticity domain. Considering the spatial information and color information of the image uniformly, an dimensional vector is formed, such that each image pixel can be represented by a vector , where represents the position coordinates of the pixel, and represents the color feature of the pixel. Let denote the mean-shift iterative formula on dimensional space, and its expression is given by the following equation:where represents the pixel value of the smoothed pixel point; represents the value of the pixel point in the square area with the smoothed point as the center and the side length is , and all the pixels in the square area are called sampling points; represents the kernel. The two commonly used kernel functions are the unit kernel function and the Gaussian kernel function; represents the weight value of each sampling point.
The main steps of crop image segmentation are as follows:
Step 1. Given initial conditions, including an initial pixel point (generally set as the first pixel point at the upper left of the image), kernel function , weight , and allowable error of each sampling point;
Step 2. Calculate the value of pixel according to equation (2);
Step 3. If , assign to , and return to Step 2; if , end the iteration of the pixel, and select the next pixel in sequence;
Step 4. Repeat steps (2) and (3) until the entire image traversal ends and obtain crop image segmentation results.
3. Identification of the Growth State of Corn and Rice Crops
Based on the crop image preprocessing results in Section 2, the CNN deep network is used to extract the image features of corn and rice crops which classify the images according to the feature extraction results. This method recognizes the canopy height and other information of crops in combination with the feature extraction results and classification results. CNN is one of important type of intelligence methods to process images very deeply.
3.1. Feature Extraction of Corn and Rice Crops’ Growth Period
Using CNN to extract features in corn and rice crop images, the dataset must have sufficient records. When the dataset is too small, the network cannot be fully trained and the advantages of CNN cannot be fully exerted [15]. This paper first uses data enhancement technology to expand the corn and rice crop images. Due to the local connection and weight sharing characteristics of the CNN, it has better distortion tolerance. Therefore, the data enhancement technology is used to expand the dataset. When the sample size is large enough, the loss of the feature information of the sample data can be avoided and the effective features of the corn and rice crop images can be extracted as much as possible [16].
In traditional CNN, saturated nonlinear functions are usually used as excitation functions, such as sigmoid function and tanh function. Nonlinear activation functions usually suffer from the vanishing gradient problem which has certain limitations. Unsaturated nonlinear functions are often used as excitation functions in current CNN structures such as ReLU functions. Compared with the saturated nonlinear excitation function, the unsaturated nonlinear function does not have the problem of gradient disappearance and they can reduce the overfitting phenomenon. In this paper, the ReLU function is used as an activation function of the convolutional network. The sigmoid function expression is shown in the following equation:where represents the saturation function and represents the output of the sigmoid function.
The unique double feature extraction structure in the CNN makes the network more capable with respect to the translation, scaling, and rotation of the corn and rice crop image sample data to a certain extent [17]. The convolution kernel size of the fully connected layer is the same as the output of the last pooling layer which guarantees the output of a one-dimensional vector. The network structure parameters used are shown in Table 1.
On the basis of the above CNN operation, the feature extraction algorithm of corn and rice crop images is improved and with the aid of the SURF algorithm [18], the geometric structure features of corn and rice crop images are enhanced along the gradient direction by times. The geometric feature vectors are in the overlapping area and the scale fractal equation of the local geometric structure of the corn and rice crop images is constructed using the following equation:where represents the image detail information function; and both represent the image structure pixel; and represent the image expansion coefficient.
Calculate the Harris corner of the corn and rice crop images, reconstruct the local area of the images according to the characteristics of the corn and rice crop images in the gray pixel area using the statistical analysis method [19], and express the image variable scale intuitionistic fuzzy set as follows according to the contour information of the images as given by the following equation:where and , respectively, represent the invariant moment feature quantities of points in different image regions and represents the ridge contrast of image feature regions. Using the Harris corner detection method [20], the Harris corner information distribution of corn and rice crop images is obtained as given by the following equation:where represents the high frequency part in the image; represents the scale factor of the Harris corner detection; and represents the normalization factor.
The manual labeling method is used to match all the sample images in blocks, and the continuous wavelet transform method [21] is used to perform the time-frequency transformation of the feature points. Wavelet transform considers continuous time signal into different scale features. The continuous wavelet transform is calculated by the following equation:where represents the wavelet transform coefficient.
The image is rotated and scaled in three-dimensional space, and the geometric dispersion of the image is obtained as shown in the following equation:where represents the transformation function obtained by the projection of the image to the space. The block wavelet transform method is used for feature adaptive matching, and the statistical features are implied in the structure and parameters of the convolutional neural network [22]. Let and , and rewrite formula (7) as given in the following equation:
Here, and both represent the image statistical feature constructor.
Combined with the template matching method, the elastic template of image feature extraction is obtained as shown in the following equation:where
In equation (11), and both represent image gradient information.
The feature analysis and contour region feature extraction are carried out in the block region of the corn and rice crop images, and the multilayer wavelet decomposition results of the image features are obtained as follows in the following equation:
Here, represents the edge information of the image; represents the image intensity information; and and represent the valley value and the peak value of the image, respectively.
The image is reorganized according to the edge, peak, valley, and intensity information of the image, and the output result of the image feature is obtained as shown in the following equation:where , , and represent image local features and and both represent image feature vectors.
The above feature extraction results are used as identification parameters to classify and identify the growth period of corn and rice crops.
3.2. Classification of Corn and Rice Crops in the Growth Period
The growth period of corn and rice crops is divided into four categories: seedling stage, jointing stage, tasseling stage, and mature stage. Based on the binary classification characteristics of SVM, it is combined with the idea of decision tree to construct a binary tree structure. Binary trees organize the imageset as per the features with specific rule. First, all samples are divided into two categories, and several easily confused categories are divided into one category, and then the above two subcategories are further divided into two low-level subcategories. Iteratively, the binary tree obtains a binary classification tree.
There are two main structures of a binary tree: one is that at each inner node, one class and the remaining classes form a partition surface and the other is the division of several categories and several categories at the inner node. The binary tree structure generation algorithm used in this paper performs clustering before classification. This method is based on the class distance method in clustering. An advantage of binary tree classification is that there is no inseparable part, and it is not necessary to traverse all the classifiers during classification.
Given the sample set , the minimum square error of dividing the sample set according to the feature extraction result using the K-means algorithm is given in the following equation:
Here, and both represent mean vectors. Formula (14) reflects the degree of closeness between different samples in the sample set to a certain extent, i.e., the smaller the values of and , the higher the similarity of the samples in the set. According to this principle, the growth period classification of corn and rice crops is realized.
3.3. Recognition Algorithm of Corn and Rice Crops’ Growth State
Based on the feature extraction of corn and rice crops and the classification of the growth period, the identification of the growth state of corn and rice crops is carried out. Canopy height measurement of corn and rice crops is a difficult point in crop growth identification. The traditional identification method is mainly that technicians go deep into the field, measure the height values of several areas, and then obtain the average value. The workload is large and the identification results are affected by the greater impact. Therefore, this paper studies the use of the fuzzy mathematics [23] model to replace the traditional method in order to improve the recognition effect.
Firstly, the growth images of corn and rice crops are collected based on laser technology, and the laser rangefinder is installed on the three-axis PTZ to dynamically scan the observation area and obtain the canopy position point set. The canopy height is obtained through internal angle error correction, triangular geometric conversion, and data fitting.
Set the horizontal scanning range as and the vertical scanning range as . Figure 3 is a schematic diagram of the rotation path of the laser rangefinder. When the altimetry device is powered on, the three-axis gimbal automatically completes initialization, the laser emission point rotates to point , the theoretical angle of the vertical angle at this position is 0°, the three-axis gimbal is at point in the horizontal position, and the theoretical angle of the horizontal angle is 0°.
Let the length matrix of the scanning point matrix be , the height measuring device starts to measure from the set initial value, and the vertical direction scans the stationary point according to the set step size. After reaching the set vertical end point, record its length matrix. ; at the same time, translate a horizontal step in the horizontal direction, and then scan to the initial vertical angle in the vertical direction, record the length scan matrix until the altimeter scans to the set end position.
The measurement height is linearly related to the scan point length, and its correlation coefficient matrix is . The value of is related to the rotation radius of the laser rangefinder and the pitch angle . In the device, is a fixed value, and is set. Assume that the height matrix of the scanning point is and is the height from the laser emission port to the crop plane after the calibration of the three-axis gimbal; its expression is given in the matrix as follows:
Due to differences in weather, sunlight intensity, etc., the collected images of corn and rice crops are significantly different. Therefore, it is particularly important to reduce the impact of environmental interference on the recognition results. Using the fuzzy algorithm to process corn and rice crop images can not only save computing time but also make the recognition results more accurate [24, 25]. The specific steps are as follows:(1)Select feature factors to construct feature sets. The feature set is constructed according to the feature vector extracted in Section 3.1. The set includes complexity, aspect ratio, mean contrast, compactness, ratio of the number of brightest pixels to the total number of target pixels, and so on. The expression of the feature set is as follows:(2)Establish membership functions and construct fuzzy sets to be identified. The key to constructing a fuzzy set lies in the determination of the membership function. The membership function is a function of the characteristic quantity which can be expressed as . The fuzzy set to be identified is as shown in the following equation:(3)Use the principle of closeness to make attribution judgment on the identified objects to complete the target identification.
The proposed work is explained in Sections 2 and 3, respectively, and the next section provides the experimental analysis.
4. Experimental Analysis
In order to verify the validity of the proposed method for identifying the growth state of corn and rice crops based on computer image processing technology, simulation experiments have been carried out to verify the results. In the experiment, the crop hyperspectral remote sensing identification method based on the random forest method and the wheat fast identification method based on GF-2 data are compared and analyzed with the proposed method. The error of segmentation results, feature extraction time, and recognition effect of corn and rice crop images are used as experimental indicators. Among them, the error of the image segmentation result affects the subsequent recognition effect. Therefore, the lower the value, the better would be the recognition results.
4.1. Experimental Samples
The experimental samples are representative real images selected from the Heilongjiang Bayi Agricultural University, Daqing. The light intensity in the region is high, and the photo light change rate is obvious. The principles of image selection are: the first is the real image of corn in the pre-growth stage which is between the three-leaf stage and the seven-leaf stage and it is called type I image and type II image, respectively. The third type is the late growth stage after the tassel stage of corn and before the mature stage; it is called type III image. According to different lighting conditions, three types of images are selected: insufficient lighting, normal lighting, and bright lighting. Figure 4 is a schematic diagram of an image of an experimental sample.
(a)
(b)
(c)
4.2. Analysis of Experimental Results
4.2.1. Error Analysis of Segmentation Results
For the sample images in Section 4.1, three methods are used for segmentation processing, and the errors of segmentation results of different methods are compared. The comparison results are shown in Table 2.
It can be seen from the data in Table 2 that the proposed method, the recognition method based on the random forest method, and the recognition method based on Gaofen-2 data segment the sample image and there are certain differences in the segmentation result errors. Among them, the segmentation result error of the proposed method is always lower than the other two. It shows a stable change trend while the segmentation result error of the recognition method based on the random forest method and the recognition method based on Gaofen-2 data is always higher than the proposed method. It is verified that the proposed method can realize the accurate segmentation of corn and rice crop images and improve the image processing effect; it is also helpful to improve the recognition effect of crop growth state.
4.2.2. Feature Extraction Time Analysis
Comparing the extraction time of corn and rice crop growth state features of different methods, the results are shown in Figure 5.
According to Figure 5, in multiple iterative tests, the feature extraction time of the proposed method is lesser than that of the traditional method. The maximum feature extraction time is only 1.4 s while the maximum feature extraction time of the recognition method based on the random forest method and the recognition method based on high Gaofen-2 data is 4.9 s and 3.8 s, respectively. The comparison shows that the proposed method has obvious advantages. This is because this method classifies the growth period of corn and rice crops before recognizing the growth state. The feature extraction efficiency is also high of the proposed method.
4.2.3. Recognition Effect Analysis
In order to more intuitively show the recognition effect of the proposed method, the recognition effect of the three methods on the sample image is further tested and the results are shown in Figure 6.
(a)
(b)
(c)
By analyzing Figure 6, it can be seen that when three methods are used to recognize the characteristics of corn image, the proposed method can eliminate redundant information. There is no problem of blurred boundary as well. However, there are many parts in the recognition result of the recognition method based on the random forest method that are inconsistent with the actual image, which increases the complexity of follow-up research. There is a problem of local ambiguity in the recognition result of the recognition method based on high Gaofen-2 data which also affects the follow-up processing effect. Therefore, the proposed method has better a recognition effect and can obtain clear and complete feature information.
4.3. Discussion
The proposed system focuses on the error analysis of error segmentation, feature extraction time analysis, and recognition effect analysis. Traditional methods are lacking of deep observation on crop development and disease recognition. Samples of images are taken in different phases, in different light, and in different environments. The proposed algorithm is able to work on all types of images which will decide an efficient way for good-quality production of crops. Random forest methods analyze different reflections of different spectra. Rapid analysis of crop images can be done with the use of different artificial intelligence techniques, which are currently used in this proposed method like CNN, decision tree, and fuzzy-based approaches. The proposed approach could be very beneficial for farmers of in the field of agriculture for analyzing the growth of crops based on automated methods.
4.4. Limitations
The proposed method is trying to overcome many problems of the traditional methods. The space complexity of the proposed method is high as it is considering the integration of many techniques which can help in the analysis of the growth of plants based on their respective images.
5. Conclusion
Aiming at the problems of long feature extraction time, large error of crop image segmentation, and poor recognition effect obtained by the traditional methods, a corn-rice crop growth state recognition method based on computer image processing technology is proposed. This proposed method focuses on getting good-quality crop by observing their images at each growth step which will help in curing diseases at that particular state. The experimental results show that the proposed method effectively solves the problems of traditional methods, and has the advantages of good recognition effect, high feature extraction efficiency, and high image segmentation accuracy. The application of this method to the field of agricultural research has certain practical significance. Different computational techniques such as CNN, binary tree, image processing techniques, K-means clustering, and fuzzy logic help get efficient results in good-quality production of crops, especially for corn and rice plants.
Abbreviations
| CMOS: | Complementary metal oxide semiconductor |
| CNN: | Convolutional neural network |
| SVM: | Support vector machine. |
Data Availability
The data used are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.