Abstract
The plant type parameters, which can be used to track and study the morphological changes of crop organs, are of great importance for breeders and testers. Based on the structural characteristics of mature rapeseed, an automatic measuring system for rapeseed plant type parameters was studied, through which the parameters of primary branches including their quantity, position, and branching angle and the geometric parameters such as plant height, main stem width, main raceme length, and the first branch height were measured. The system hardware includes a manual cutting-short device, an image acquisition darkroom, and a control circuit, used for capturing videos of a stably rotating rapeseed plant in a darkroom with an adjustable light source. The system software is used to connect and control CCD camera and other hardware and run specific image processing algorithms to process and analyze videos to obtain plant type parameters. The human-machine interface provided by the system can be used to set working parameters and display and process measurement results. To evaluate the accuracy of this prototype, batches of rapeseed plants were tested. The results showed that the mean absolute percentage error (MAPE) for the system was about 2.74%, and the automatic measurements had a good agreement with manual measurements. The measuring efficiency was approximately 2.1 times over manual method. In conclusion, the system, which automatically extracts plant type parameters with high throughput and high precision, is of practical value as it can effectively reduce the labor intensity of researchers and provide important basic data for the research of rapeseed breeding and agricultural machinery design.
1. Introduction
In recent years, with the completion of whole-genome sequencing of a variety of plants, the application of plant phenomics to the full tapping and analysis of genome, transcriptome, proteome, and other omics information is of great significance to improve plant breeding efficiency [1]. As one of the directions of genetic improvement, crop ideotype breeding can improve the growth and development of the plant under dense planting conditions by adjusting the plant type, which is also one of the decisive factors to achieve the mechanization of crop production [2]. Most plant type traits have a notable genetic correlation with single-plant yield and traits relative to the yield [3]. Plant height, branch position, branching angle, and main raceme length affect the rapeseed yield indirectly by influencing yield factors or crop lodging [4]. As an important phenotypic trait in the analysis of the phenotype of breeding crops, plant type parameter is a general concern in genetic breeding and agricultural production. In particular, characteristics of rapeseed plant types are quite different from those of rice and wheat, which results in a high loss rate in the mechanized harvesting. The cooperation of agricultural machinery and agronomics is the best way to resolve this problem by improving the coupling degree of the harvester’s parameters and rapeseed plant type parameters [5]. The quantitative measurement of plant type is not just beneficial to cultivate and breed more rapeseed breed with diverse plant types but also useful for optimization of the harvesting machinery design.
With the advantages of low cost, high efficiency, and easy popularization, the phenotypic detection based on visible light images can not only obtain plant growth images in situ, in real time, and continuously, but also resolve various phenotypic parameters such as plant structure, morphology, color, and texture [6]. The 3D reconstruction technique is widely used for phenotypic detection because of its high precision. For example, Zong et al. proposed an improved algorithm based on skeleton extraction, and PMD depth camera was used in the field environment to measure rapidly and nondestructively the three plant type parameters such as plant height, stem diameter, and leaf inclination angle [7]. Using 2D skeleton matching to generate 3D skeletons and artificially interactively fitted spline curves, Zhao et al. achieved a 3D skeleton extraction and reconstruction method for corn plants based on the theory of stereo vision [8]. Paulus et al. proposed a high-precision plant type parameter measurement method based on 3D laser scanning [9]. Fang et al. proposed a method for high-throughput 3D wheat PA study, and the plant height and other phenotypic parameters were obtained from multiple images to reconstruct the plant volume and estimate the plant fresh weight [10]. Compared with the time-consuming 3D reconstruction technology that needs large amount of calculation, the phenotype detection based on machine vision technology is more widely used. Kun et al. proposed an image processing method to measure panicle traits such as the number of awns, average awn length, panicle length, and panicle type, and the relative errors of the system measurements were all less than 4% [11]. Zong et al. proposed a new skeleton extraction method, which improved the precision of plant phenotypic parameters such as plant height, leaf length, and leaf inclination, especially the precision of plant type parameters [12]. Sun et al. applied computer image processing technology to the automatic measurement and analysis of the stem growth angle parameter of tomato plants [13]. Li et al. improved the image segmentation and skeleton extraction algorithm to extract the wheat plant skeleton and then calculated the plant height of wheat with the minimum rectangle method, the leaf length with the pixel look-up method, the leaf base angle and opening angle with Hough transform, and the detection accuracy which was all over 90% [14]. Huang et al. developed an automatic measurement system for the panicle length of rice based on dual-camera imaging, with an average relative error of 1.23% [15].
Studies were made on the plant type parameters of rapeseed at seedling stage. For example, Xiong et al. established a high-throughput stereo imaging system for the reconstruction of 3D canopy structures of rape seedlings with leaf area and plant height and 19 individual leaf characteristics of leaves were also extracted [16]. The 3D reconstruction of rapeseed at the seedling stage was made by Shi et al. with monocular vision and laser scanning technology, and 4 plant type parameters including the plant height, petiole length, blade length, and blade area were measured in the reconstruction model [17]. It is difficult to use 3D reconstruction technology to achieve the rapid detection of plant type parameters due to the limits of its technical characteristics, as the rapeseed at mature stage has a very complex plant structure. A feasible idea is to study the targeted machine vision technology. For example, Wang et al. developed a method for measuring the branching angle of rapeseed and the angle of siliqua, which is of high repeatability, precision, and operability. However, this method involves a number of steps such as sampling, photographing, path drawing, and angle value extraction, which require tedious manual work, thus making it less efficient [18]. In general, high-throughput and accurate measurement of the plant type of rapeseed at mature stage is badly in need. This paper studies a mechatronic device based on machine vision technology for agricultural research, which is a suggested lab setup but, only by simple manual work, is able to measure the trait parameters of rapeseed plants, including plant height, canopy amplitude, main raceme length, main stem width, the height of the first lateral branch, and the quantity, position, and angle of the primary branches. With high accuracy and at a high automatic level, this device will provide reliable data support for the genetic breeding of rapeseed.
2. Materials and Methods
2.1. Experimental Materials
All the materials were collected from the experimental field of College of Plant Science, Huazhong Agricultural University in May 2017. A total of 52 Brassica napus plants from 13 varieties at the harvest time were collected, including the Zhongshuang No.6 and Huayouza No.62. Firstly, a small number of possible petioles were manually removed from the main stem of the plant. The rapeseed plants were then fixed on a self-made image acquisition device, and the CCD (model SUNWAY U3D500C-J, 5MP) was started to record a video of the plant rotating for one week. By adjusting the camera’s frame rate and the speed of the turntable, a varying number of video frames were obtained. 30 to 120 frames are general rates that were set to satisfactory measurement accuracy.
2.2. Automatic Collecting Device for Videos of Rapeseed Plants
2.2.1. Design of the Overall Structure
With a large overall size and size fluctuation, the rapeseed plant is heavy and with a high center of gravity. So, it is necessary to design a special electromechanical device to clamp and rotate the plant to complete the video capture of 360-degree plant rotation. The structure of the device is shown in Figure 1(a), the model of which is shown in Figure 1(b). The image acquisition device consists of three parts, including a manual cutting-short device, an image acquisition darkroom, and a central control system.
(a)
(b)
(1) Manual Cutting-Short Device. For extremely tall plants, a certain length of stalk needs to be cut off for measurement. The cutting-short device is supported by a pair of sliding rails, the height of which can be adjusted according to the height of the rapeseed in a single measurement. Accordingly, this height should be entered once in the human-machine interface of the computer to compensate for the measurement error of the shortened plant height. A series of spacing holes are designed in the cutting-short device to make it easy for the rapeseed to accurately and vertically insert into the bottom. A pair of fruit shears is fixed at the top of the spacing hole, and a trigger switch is also installed. Manual use of the fruit shears will be recorded as a cutting behavior and delivered to the electric control box.
(2) Image Acquisition Darkroom. The darkroom is a rectangular 3D stand made of aluminum alloy, with a double-edged door that can be electrically controlled to close. The other sides are sealed with black light-absorbing cloth, and four LED lights are installed at the top as the lighting source. An industrial CCD camera is installed on one side of the darkroom to capture video or images of the rapeseed plant. A shaft bracket with double fixtures is used to fix the rapeseed plant, which is installed on an electric turntable and then fixed on a linear guide rail as a whole. Clamped by the shaft bracket, the rapeseed plant is sent by the linear guide rail from the outside of the darkroom to the interior and then rotated by the electric turntable.
(3) Central Control System. The first part is a control circuit based on the STM32 for controlling stepper motors and other electrical elements. The second part is a computer for coordinating and controlling turntables, cameras, and other hardware to run image processing algorithms and provide human-machine interface.
2.2.2. Human-Machine Interface Software and Its Operation Procedures
LabVIEW 7.0 was used to design system software running on a computer to connect and control hardware such as CCD and motor control circuits and to run image processing algorithms and output results. The software allows the operator to easily and conveniently measure rapeseed plant parameters. The human-machine interface (HMI) we developed is shown in Figure 2. After starting the system software, the measurement system needs to be carried out as follows.
(1) Cut Plant Stems (as Needed). If the length of the stalk is higher than the image acquisition room as is visually measured by the operator, it needs to be cut short. Insert the rapeseed into the spacing holes of the cutting-short device until it reaches the bottom, drop a fixed height (typically 50 cm), and manually close the fruit shears to cut off the rapeseed stem. At the same time, the closing action of the fruit shears will trigger the limit switch, and the trigger signal will be stored by the control circuit. When processing the data, the system software will automatically read the stored value to correct the plant height parameter.
(2) Fix Plants. The treated rapeseed plant was inserted into the shaft bracket by the operator. The middle of the stem was passed through the center of three pieces of roller elastic fixture, and the lower part of the stem was inserted into the three-jaw fixture on the electric turntable. Press the “Start Holding” button, and the linear guide rail will push the shaft bracket into the darkroom until the limit device is locked. Then, two fixtures will clamp the plant and fix it to the shaft bracket. The door of the darkroom will be automatically closed.
(3) Shoot a Video. Press the “Start Shooting” button, and the electric turntable starts to work, driving the plant rotating. Delay for 5 seconds till the plant rotates steadily. After receiving the trigger signal, the PC will call CCD to shoot the rotation video of the rapeseed and save it to the specified hard disk path.
(4) Start Image Processing Program. Specify the video save path and file name and click “Start Program,” and LabVIEW system program will call MATLAB function to analyze and process the video. The measurement outcome will be displayed on the human-machine interface and saved to the excel form as well.
2.3. Image Processing Algorithm for Plant Type Parameter Measurement
The flowchart of the image processing algorithm for plant type parameter measurement is shown in Figure 3. Separate image frames were obtained by resolving the video. Then after filtering, skew correction, and size conversion, a preprocessing image was obtained. The minimum enclosing rectangle of the plant in the preprocessing image was calculated to obtain the geometric parameter of the plant. A series of treatments such as cutting stems and removing siliqua were performed on the preprocessing image to calculate the quantity, position, and angle of the primary branches.
2.3.1. Image Preprocessing
At first, the video obtained in the shooting needs to be resolved into separate frames. As the background of the image is a black screen, which involves color difference and impurities, filtering and other processing need to be carried out to remove noises. The unit of measure for the measurement results is pixel, which needs to be converted to physical units according to the calibration parameters. In addition, due to the influence of gravity, there is an inevitable skew of the rapeseed plants that affects the measurement of certain parameters, which, therefore, needs to be corrected. The process and the results of each step of the image preprocessing are shown in Figure 4, including the following three aspects.
(a)
(b)
(c)
(d)
(e)
(1) Image Filtering and Binarization. Convert the original color image to a grayscale image and carry out a 5 × 5 median filter. Then, the OTSU algorithm was used for binarization to obtain a binary image with a pure background, as is shown in Figure 4(a).
(2) Calibration of Conversion Scale. A picture of a ruler needs to be taken as a calibration object before the measurement system works for the first time. As the gray values of the ruler and black cloth vary greatly, the overall area is larger. After using the structural elements of 9 × 9 to open the binary image, most of the rapeseed plants were split or eliminated, while the ruler area almost remained unchanged. Calculate the connected domain in the image after the erosion, the largest of which is the one corresponding to the ruler. Extract the minimum enclosing rectangle of the connected domain, the width of which is the pixel width of the ruler. And calculate the conversion scale based on the physical length of the ruler.
(3) Skew Error Correction. The main stem refers to the cylinder-like part below the primary branch of the plant, whose skeleton is an approximately straight line. Calculate the principal axis equation of the main stem for the correction of plant parameters. Set the resolution of the image to be checked to X × Y. First skeletonize the image and then traverse the skeleton image row by row from the top to the bottom, and search for a row with only one black pixel, which is the position of the primary branch. Intercept the part below the primary branch position of the original image and calculate the principal axis equation of the connected region. The process is shown in Figure 4.
If the image is treated as a binary bounded function f(x, y) with a two-dimensional density distribution, the function value f(x, y) represents the gray value of the image pixel at the point (x, y). The (j + k) moment of the dimension density distribution function f(x, y) is expressed as Mjk, as shown in the following equation:
Firstly, the second-order center distance is used to calculate the principal axis direction angle of the image, as shown in the following equation:
In the formula, μ11, μ20, and μ02 are center distances of the (1 + 1) order, (2 + 0) order, and (0 + 2) order, respectively, and the computing formula is given by
Secondly, calculate the centroid () of the only connected region in this part of the image:
Finally, calculate the principal axis equation of the connected region from tanα and the centroid coordinate.
In order to correct the measurement error caused by the plant skew, the original rapeseed image is rotated clockwise by α angle and is bilinearly interpolated, followed by subsequent processing and calculation.
2.3.2. Quantity and Positioning of Primary Branches
Only by dealing with many frames of images and subsequent screening can the exact value of the parameters like the quantity and angle of the branches be obtained. It can be divided into the following four steps.
(1) Plant Cropping. Above the highest branch of the rapeseed plant grow dense branches and siliqua. According to this feature, we are able to automatically cut the plant image to retain the primary branch and remove the other parts. A skeleton image of rapeseed plants was obtained by cutting off 3/4 of the main stem and refining the remaining images. Use a full 1 template that is slightly larger than the width of the main stem (k takes 1–1.5) and move from the bottom to the top along the principal axis to perform multiplication. With () pixels moved each time, the number of pixels that have a gray value of 0 (the skeleton point) in the template effect is counted. With the completion of the traversal, the statistical results of the number of pixels that the template had effects on are received. Within the scope of the template, if there is only the stem, the number of pixels is equal to about . If there are large branches, the number of pixels will sharply increase and then decrease, which presents a plane-convex trend; if entered into the section above the highest primary branch, the changes of the polyline become irregular because of the disturbance of the skeleton of numerous branches and siliqua. Calculate the mode (the most frequent occurrences) of all the result data in a traversal and identify the stem segment without branches. If the mode occurs frequently, it means that the template is below the highest branch; if the mode occurs infrequently, it indicates that the template is above the highest branch. Record and analyze the density and position of the occurrence of the mode and cut off the parts above the highest branch. The specific process is shown in Figure 5.
(2) Removing of Siliqua and Branches. In the automatically cropped skeleton image, there still remained a small portion of siliqua or branches that needed to be further removed. Most siliqua or branches grow obliquely upwards at about 45 degrees on the stalk. We can remove them by this feature. Firstly, the morphological operator [0 1 0; 0 1 0; 0 1 0] was used for expansion to enhance the skeleton of the stem. Then, the morphological operators [0 0 1; 0 1 0; 1 0 0] and [1 0 0; 0 1 0; 0 0 1] were used for corrosion to separate a large number of branches and siliqua. Minus the corrosion results from the skeleton images and then remove the small area of isolated connective domain. Finally, a circular structure element with a radius of 1 is used for expansion and dot product with the skeleton image to connect the branches with the pixels of the main stem in the image to obtain the images with notably removed branches and siliqua. The process is shown in Figure 6.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(3) Detection of the Quantity and Position of the Primary Branches. The length of the primary branch is significantly greater than that of the interfering components and away from the stem as well. This feature was used to identify the primary branch. First, set the gray value of the area that puts the principal axis of the stalk as the center line and has a width of to 255 and a skeleton image without stalks can be gained. Detect all the connected domains in the image and calculate the minimum enclosing rectangle of the connected domains. Then calculate the length of the diagonal of the rectangle and its distance from the principal axis. Set appropriate length and distance threshold and screen them, and the connected domains that meet the requirements represent the primary branch area. The recognition process and results are shown in Figure 7. The red rectangles are the enclosing rectangles of all the connected domains detected and the green ones are the final screening results.
(4) The Included Angle of the Primary Branch. The linear equation of the primary branch was obtained by extracting the pixel of the corresponding connected domain of the primary branch near the 1/3 of the stem and performing RANSAC linear fitting. The solution of the included angle between the principal axis equation of the main stem and the fitting linear equation is the included angle of the branch and the main stem. Since the angle of photographing images affects the included angle between the primary branch and the main stem, only when the plane formed by the fitting line and the principal axis is parallel to the CCD plane, the included angle can be the largest and be seen as the angle at which the branch was born. The images of each frame of the rapeseed rotating video are processed to obtain the included angle value of the branches and the main stem of the image sequence, among which the maximum output value is the angle at which the primary branch was born.
2.3.3. Measurement of Geometric Parameters
After filtering and threshold segmentation, the binary image was obtained from the corrected original rapeseed plant image, which is shown in Figure 4(c). Then calculate the minimum enclosing rectangle of the plant in Figure 4(e), whose length is the plant height and whose width is the canopy amplitude . The corrected binary image is traversed from the top to the bottom row by row to search for a row with only one black pixel. The pixel coordinates of the row is , and the primary branch height is as shown in the following equation:
The main raceme length can be calculated according to the position coordinates of the highest branch and the coordinates of the highest first lateral branch , as shown in the following equation:
According to Figure 4, the part below the primary branch of the rapeseed plant is the main stem. Test the connected domain, traverse it from the top to the bottom row by row, count the difference value of the abscissa of the two sides of each row, and then calculate the average of all the differences; this is how we get the width of the main stem in the image. The measurement of geometric parameter of the plant type is as shown in Figure 4(e).
3. Results and Discussion
Compared with the automatic measurement in the system, manual measurements were performed by two skilled workers, and the mean measuring result was calculated as ground truth. The following procedures are the details of manual measurements. The quantity and position of the primary branches in the image can be acquired by human eyes, and their heights can be measured by a ruler and a soft string. The primary branch angle was obtained by measuring the angle between the main stem and the primary branch stem with a standard protractor. The measurement of geometric parameters was carried out by a ruler and a soft string. For example, stretch the soft string along the main raceme stem to both ends and measure the length of the straitened string with a ruler when measuring the main raceme length. To validate the system measuring accuracy, the root mean squared error (RMSE) defined by equation (7) and the mean absolute percentage error (MAPE) defined by equation (8) for the plant type parameters measurement were computed, respectively. In addition, a large number of rapeseed plant samples were also tested under the condition of continuous measurement, and the time costs of manual measurement and automatic measurement were compared to test the system’s efficiency:where xai represents an automatically measured value by the system and xmi represents a manually measured value.
3.1. System Measuring Accuracy
To evaluate the system measuring accuracy, 52 plant samples were used to eliminate the measuring error in geometric parameters. Besides, a total of 24 plants from 6 rapeseed varieties with significant plant type compactness differences were tested to evaluate the measuring accuracy in detection of the primary branches. The plant type parameters results measured by the system and manual method (as previously mentioned) were compared. The evaluation results of the system measurement for the above plant samples are shown in Table 1 and Figure 8. The results showed that MAPE and RMSE results were 3.42% and 0.31 for the quantity of primary branches, 5.62% and 2.16 degree for the first branch angle, 1.21% and 14.15 mm for plant height, 3.25% and 0.62 mm for main stem width, 2.15% and 13.16 mm for canopy amplitude, 2.10% and 10.55 mm for main raceme length, and 1.43% and 7.62 mm for the first branch height, respectively. The results proved that automatic measurements are in excellent accordance with manual measurements and it was concluded that the measurement system was generally performed with high accuracy, an average MAPE of 2.74%.
From the results, the measuring error for geometric parameters was mainly resulted from the fact that the algorithm used in this paper directly calculates the linear distance without considering the errors caused by the severely curved plant structure, which needs to be improved in the subsequent algorithm. The results showed that two reasons led to the error occurred in the testing of the quantity of primary branches. For one thing, during the preparation phase, the main stem was not completely manually trimmed so that the siliqua or small branches which were not removed were mistakenly tested as primary branches. For another, 45- and 135-degree templates were used in the algorithm of testing primary branches. When the plant type was too compact or not, the angle between the primary branch and the main stem far exceeded the template angle, resulting in the poor removal effect of siliqua and small branches and thus leading to a detection error. This error can be eliminated by manually observing the compactness of the plant and adjusting the template angle in real time. Meanwhile, the results also showed that there was an error in the measuring accuracy for the primary branch angle because of the 360-degree rotating video of the plant used in the test. When the sampling frequency of the rotation angle was limited, the plane formed by the branch and main stem was not parallel to the CCD plane, which led to errors. This error can be reduced by decreasing the angular sampling interval. In this way, however, the time for image acquisition and calculation would be increased. But the results also proved that the system had a good performance for these plant types in general.
3.2. System Measuring Efficiency
Accordingly, the manual measurement for each plant was performed with following procedures: identifying the neck of the primary branch, counting the number, measuring the geometric parameters, and recording the results, and the measurement time for each plant is the sum of time costs of the above procedures [15]. All the measurement was performed continuously. Meanwhile, the time costs of manual measurement and automatic measurement were both recorded. It took approximately 75 minutes to measure the 24 plant samples by using the system. By contrast, it took nearly 160 minutes for a skilled worker to complete the measurement manually. Thus, the system was almost 2.1 times as efficient as the manual approach. The experimental results indicated that it took about 190 s for the system to measure a panicle. Generally, it took approximately 20 s to place the plant and take it out and then begin the next measurement. It took about 60 s to acquire the video of the rotating plant. The parameters were extracted in approximately 110 s on a workstation configured with a 3.1 GHz main frequency, 8 GB of memory, and 4 CPU cores. The manual operation of the system (placing the plant and taking the plant out) was still time-consuming, which accounted for approximately 11% of the measurement time of the system. Moreover, manual operation of the system was tedious in the continuous-measurement mode. To improve them, an automatic plant transport platform might be incorporated in the practical promotion.
4. Conclusion
This paper describes a new prototype for the efficient and automatic measurement of rapeseed plant type parameters. A camera imaging was employed to provide a plant video for analysis. Equipped with dedicated software for system control and image analysis, this system was able to accurately identify the position of the primary branches and finally extract geometric parameters with high accuracy, regardless of the plant type. Therefore, this system could be applied in research fields related to rapeseed such as seed selection, breeding, and agricultural machinery design. As the system is easy to operate, it can effectively reduce the work intensity of agricultural researchers. To conclude, providing a novel tool for nondestructive testing of rapeseed plant type parameters, the system, which is suitable for the identification of rapeseed materials on a large scale such as large population genetic analysis and breeding identification, is quite advanced and practical.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by the Fundamental Research Funds for the Central Universities (no. 2662017JC043) and Innovation Training Plan of University Student (no. 201710504059).