Abstract
Exact enumeration of aphids before the aphids outbreak can provide basis for precision spray. This paper designs counting software that can be run on smartphones for real-time enumeration of aphids. As a first step of the method used in this paper, images of the yellow sticky board that is aiming to catch insects are segmented from complex background by using GrabCut method; then the images will be normalized by perspective transformation method. The second step is the pretreatment on the images; images of aphids will be segmented by using OSTU threshold method after the effect of random illumination is eliminated by single image difference method. The last step of the method is aphids’ recognition and counting according to area feature of aphids after extracting contours of aphids by contour detection method. At last, the result of the experiment proves that the effect of random illumination can be effectively eliminated by using single image difference method. The counting accuracy in greenhouse is above 95%, while it can reach 92.5% outside. Thus, it can be seen that the counting software designed in this paper can realize exact enumeration of aphids under complicated illumination which can be used widely. The design method proposed in this paper can provide basis for precision spray according to its effective detection insects.
1. Introduction
As the main representative of crop pests affecting wheat field and orchard fruit output [1], aphids with a huge number are widely distributed. By hanging the yellow sticky board on plants, people can take count of the aphids on the board so that they can determine if pesticide is needed. However, this method is imprecise.
Compared with manual counting, machine vision can calculate the number of pests objectively, accurately, and effectively through analyzing the image information. To solve these problems, a lot of research has been carried out.
Cassani et al. approached automated EEG-based Alzheimer’s disease diagnosis using relevance vector machines [2], Chakraborty and Newton studied climate change, plant diseases, and food security [3], Pal and Foody used evaluation of SVM, RVM, and SMLR for accurate image classification with limited ground data [4], Patel et al. used improved multiple features based algorithm for fruit detection [5], Luo et al. develop an aphid damage hyperspectral index for detecting aphid (Hemiptera: Aphididae) damage levels in winter wheat and so on [6].
Yue-hua et al. [7] extracted the aphids by using region growing algorithm and OSTU threshold algorithm in the natural environment, and the recognition rate of vector machine training images can reach 90.7% by using -means; Bai-jing et al. [8] extracted the aphids on the use of the method of G component threshold (each pixel in the aphids and leaves) and carried out counting research on the use of connected area markers. Jingxiao [9] transmitted the image of the pest to the PC side for HSV color model segmentation and then the pest is identified by BP neural network. Vincent et al. [10] proposed a target detector that uses the analytical decision support system to make the detector adapt to changes in the plant to identify aphids on the plant; Tirelli et al. studied the remote monitoring system to prevent pest outbreaks and trigger an alarm when the number of pests is too high [11]; Ukatsu achieved the remote counting function of rice margin bugs through using the wireless network technology.
These methods reflect the following problems: they will be influenced by illumination and the threshold is not unique in threshold extraction method. Extracting aphids by PC remoting method costs too much network traffic with poor timeliness. Therefore, to obtain images in the complex environment and high impact of the scene, pests can be identified and counted without relying on the feedback of the PC side.
2. Materials and Methods
The acquisition of experimental images was obtained at the West Campus of Agricultural University of Hebei; image acquisition equipment is Samsung Galaxy 4 phone that has a Qualcomm eight-core processor with Android 2.3 system, API 25. It is a common type with satisfactory parameters. The chapter contains two parts of picture distillation; in the first part, the GrabCut algorithm [12] is used to extract yellow sticky board image from complex background and correct it to the same size and position by perspective correction method in order to extract aphid image [13]; in the second part, the method of weighted average gray scale is used to make the yellow sticky board image preprocessing, single image difference method is used to eliminate the influence of illumination, and OSTU threshold segmentation and morphological erosion method is used to extract yellow sticky board image, laying the groundwork for the subsequent counting program.
2.1. Extraction the of Yellow Sticky Board Image
GrabCut is an improved algorithm based on GraphCut. GrabCut is an algorithm that uses the RGB three-channel hybrid Gauss model for color segmentation; segmentation can be completed by using region pixel set which contains background. The algorithm belongs to iterative segmentation, and the segmentation accuracy is higher. Gibbs energy function is a specific vocabulary that means a formula that can calculate the weights and cost.
is a method when the target pixels are classified as foreground or background models, which are obtained by matrix iteration of classifying and rewriting pixel:
is the boundary energy smoothing term, when the boundary pixels are discontinuous; it will do some boundary pixel classification processing:
makes the boundary energy more and more little by iterative method; after iteration, the parameters of the Gauss mixture model of target and background are higher.
2.2. Correction of Yellow Sticky Board
Because the shooting angle is not fixed, the positions of yellow sticky board which has been extracted are not fixed. In order to improve the accuracy of aphid identification and counting, the yellow sticky board must be corrected to the same size and the same position when a target image has been segmented; getting the elevation view, the aphids can be counted accurately.
Transformation from distortion to elevation view can be completed by using perspective correction method [14] which uses the four vertexes of distortion image and elevation view as the correction basis. The top left vertex is the key of correction. Firstly, the corrected image matrix is obtained by using the distorted image matrix ; you can get the top left vertex by using the relationship between . Secondly, according to top left vertex, the length and width of the rectangle can be used to obtain the information of four vertices.
Figure 1 is the elevation view of yellow sticky board by using GrabCut and perspective correction method. We can see that the picture has a clear boundary and regular shape which will make the aphids extracted easily.
2.3. The Image of the Yellow Sticky Board Preprocessing
The color image change into gray image can reduce amount of computation in the image processing process and reduce the complexity of the calculation [15, 16], the gray scale value obtained by the weighted gray scale method is more in line with the perception of the corresponding gray scale of each pixel, and the brightness is moderate; it is conducive to the next aphid image extraction.
2.4. Aphids Extraction
2.4.1. Image Difference
Due to the influence of light, even if more than two thresholds segmentations are proposed; each of the pictures taken in the natural environment has an uncertain gray value, so the threshold is uncertain too. Aphid color is relatively single, so the impact of light is mainly reflected in the background; an image can be seen to consist of foreground and background; when the background is removed, the influence of light is removed. Regarding the process of image difference, firstly the original gray image will be taken for fuzzification as SRC and then subtract the fuzzy background DST. The difference that results, DIFF, is the aphids image.
2.4.2. Fuzzy Background
The gray scale image after blurring is compared with the gray scale distribution in the original gray scale graph, and if the local gray scale distribution is equal, it can be used as the background image. Compared with the fuzzy effect of the three filters, the Gaussian weight filter has the smallest noise, which is consistent with the original gray scale distribution. When the kernel value is (51, 51), the background blurring is the best.
2.4.3. Threshold Selection
Image difference is only gray image, and the gray image is not the result of image segmentation and needs the threshold segmentation [17]. A large number of aphids are missing counted by using binary threshold method, while there are too many contour break points by using self-adaption threshold method. So this paper uses the OSTU threshold segmentation method which has a clear and integrated contour as the binarization method.
2.4.4. Morphological Operation
In the natural environment, the aphids on the yellow sticky board have a high base and a high density, and some of the aphids’ adhesion does not affect the overall counting result and requires some corrosion of the partially adhered aphids to reduce the error. This paper uses the elliptical nucleus with a nuclear value of for corrosion operation. This method has a good effect on elimination of adhesions.
Figure 2 shows the use of image difference algorithm to obtain the aphid images; aphid images are clear and complete.
2.5. Aphids Count
2.5.1. Feature Extraction
In order to count accurately, it is necessary to identify whether there are aphids. With access to relevant information we can see that the aphid’s length is between 1.5 mm and 4.9 mm. Most aphids are 2 mm in length, and there are differences in the lengths of other insects in the orchard; for example, the apple leaf mites are about 0.3 mm. So distinguish between aphids and nonaphids according to area characteristics. First, all the contours in the binary image are identified as digital marks, and the sum of the pixels in each connection domain is calculated to find the relationship between the actual length of the aphids and the pixel values, as shown in Figure 3.
According to Table 1, based on the relationship between the pixel values in each connected domain and the actual lengths of aphids, we can draw a conclusion that when the pixel value is less than 20, the insect’s body length increases by 0.1 mm and the pixel value is increased by 1 px~2 px; when the pixel value is higher than 20, the insect’s body length increases by 0.1 mm and the pixel value is increased by 3 px~4 px. In this paper, the aphids involved are mostly 1.5 mm to 2.8 mm, so when the aphid area characteristics range from 15 px to 48 px, nonaphids range from 1 px to 10 px is the most suitable. According to the range of aphid area characteristics derived from the above study to identify aphid images on the yellow sticky board [18], in the obtained result graph, the aphid image is marked with white and nonaphid images are not marked with white, consistent with the human eye.
2.5.2. Aphids Counting Method
Using findContours() contour extraction method to find all the contours of the image. Firstly, the outline which can be detected in the image will be converted to vector form to iterate, and then the aphids section will be screened out by using “for-cycle” according to their area features [19].
2.6. Design and Implementation Based on the Android Platform
2.6.1. Software Overall Flow Chart
In order to meet the needs of the public for the convenience of equipment, Android system as a mobile platform mainstream system [20] in the mobile side of the software development plays a guiding role; based on the Android platform aphid count APP will greatly facilitate the majority of farmers and agricultural science and technology workers. The aphid counting software in this paper uses the Java programming language in the Android studio platform with OpenCV computer vision library development for Android 4.0.3 and above. The software implements the counting of aphids; the whole is divided into three modules: image acquisition module, image processing module, and image counting module; as shown in Figure 4, now we design and implement the part of the content of the software.
(i) Image Acquisition Module. The software obtains the picture in two ways: invoking the local camera to shoot in real time or retrieving the existing photo in the local album. After the photo is taken to ensure the quality of the picture, the appropriate image compression can not only reduce the computational complexity of the image processing process but also avoid the overflow caused by the image being too large to cause the program to crash. First create an intent object and set the action of the object as the string of MediaStore, ACTION_IMAGE_CAPTURE. Then use the string of startActivityForResult to start Intent program. After setting the length and width, the image information will be loaded into Imageview by using decodeFile method. At last the image information will return and be displayed on the next interface.
(ii) Image Processing Module. In the image processing module, we set “yellow sticky board segmentation” “yellow sticky board correction” “aphids identification” to extract the aphids.
Using GrabCut algorithm to extract yellow sticky board, first create a Rect object to create a rectangular box to select the possible prospects in the image and use the canvas.drawRect () method to draw the rectangle and pass into the mask matrix. Create a mask object as the display position of the segmented yellow sticky board image, which is the image of the CvType.CV_8UC1 channel that needs to be of the same size as the original image. Black is used as the color of scallar which means the background of output image is black as well. Finally the segmentation can be completed by calling the arguments.
Use the perspectiveTransform () method to correct the yellow sticky board. Use the “for-cycle” to traverse the four edge vertexes of the two intersecting lines in the segmented image by using GrabCut. Then compare the coordinates of the four vertexes with the midpoint of the yellow sticky board. For example, in the coordinates of the horizontal axis, the value of the vertical axis is less than the corresponding value of the midpoint; then the point is the top left vertex. According to the upper left, upper right, lower left, and lower right of the order of the four vertex coordinates in the variable called startM, the perspective transformation method can be corrected.
Use single image difference method to extract aphids. Through the Imgproc class cvtColor, color image will be converted to the output image grayMat gray image. Set the parameters of nuclear value Size as Size (51, 51); then use the Imgproc.GaussianBlur method to complete the process of fuzzy background. Use Core.absdiff() for differential operation. Use the Imgproc.THRESH_OTSU for threshold operation after difference background image. The differential images need to undergo morphological corrosion to remove the effects of the aphids. Set the morphological action as MORPH_ERODE and set the shape of structure element kernel1 as MORPH_ELLIPSE with a new Size (5, 5) by using getStructuringElement(). Elimination of adhesions will be completed.
(iii) Image Counting Module. The image counting module filters the contours based on the total pixel range in each connected domain and stores the count on the counter. The output value is the number of aphids.
The counting process can be seen as a traversal of the “funnel”; then we should define two containers: the first container is stored in all the contour data, whether or not meeting the characteristics of the requirements, and the second container is stored after the screening of the contour information. Traverse the contour type to Imgproc.RETR_CCOMP and output it as Imgproc.CHAIN_APPROX_SIMPLE. The “for-cycle” is the key of selecting. Set 15 px to 18 px as the shape of aphids image in the connected domain and set scallar (255, 255, 255) to show aphids as white.
3. Results and Discussions
Using the aphid counting software to count the images in greenhouse and the images in orchards, Figure 5(a) shows the counting results of the aphids in the greenhouse, and Figure 5(b) shows the counting results of the aphids in the orchard.
(a) In greenhouse
(b) Orchard
Based on the least squares method, a linear regression analysis has been carried out by taking 10 sets of data from Table 2 in the greenhouse to get the fitting curve. -axis is the APP counts number and -axis is the manual counts number. As shown in Figure 6, the relationship of the fitting between the mobile APP count and the manual count is ; the fit degree factor is up to 0.99504, indicating that the aphid recognition and mobile APP counting can be used to achieve functions of recognition and counting. Adding two sets of orchard data from Table 2 into the coordinate system, we can find that the two points are in the vicinity of the curve, indicating that the method is basically effective. Because the orchard environment is more complex, the accuracy rate of the method is slightly worse than the greenhouse; as shown in Table 2, the greenhouse accuracy rate is up to 95% and orchard accuracy rate is up to 92%. In general, the aphids identification and counting method is highly reliable.
4. Conclusion
In the past, aphid identification focuses on the separation of a single kind of aphids, few on different kinds of pests. The count number is less, and the aphids’ antennae, foot, and wings need to be extracted precisely, because the operation complexity is high; most of the application software that implements the aphid recognition and counting function is concentrated on the PC side. The traditional counting APP collects the photos through the Internet to transfer data by the PC processing, and then the results return back to the mobile App, the process Overreliance on the PC side and the Internet; its real time performance is poor. Collection of aphids requires special equipment; this professional equipment is not convenient and also very expensive. The device that collects the aphids is not universal.
This paper uses yellow sticky board in the natural environment to attracts aphids and some other pests, and collects images in the real growth environment of plants. Aphid counting software overcomes the complex natural environment and the impact of light caused by image processing difficulties; it does not rely on the PC side and the network feedback; it can quickly identify and count aphids, which are the representative of the pest group; it has very practical value. The accuracy of the aphid counting software is compared with the result of manual counting; the accuracy rate of the aphid counting software is more than 95% in the greenhouse environment; the accuracy rate of the aphid counting software is more than 92%. The operation is smooth and convenient. The method proposed in this paper can extract and correct the image of yellow sticky board, eliminate influence of illumination on the image, and count aphids accurately.
Due to the limitations of the season and the study site, the aphid counting software needs to design a memory storage module to achieve the same number of yellow sticky insect boards to make trend prediction function.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work is supported by the new facilities of agricultural machinery research and development (17227206D).