Abstract
Bruise may cause spoilage, reduce commodity economic value, and give rise to food quality and safety concerns. Therefore, it is crucial to detect whether a loquat is bruised and when it is bruised to save storage and transportation costs. At present, the bruise of loquats is mainly discriminated by the operator’s naked eye, which is affected by personal habits, light intensity, and subjective psychological factors. The detection method is time-consuming, inaccurate, inefficient, and difficult to identify the bruise’s time of loquats. Due to the fact that the color features can be used to perform the conditions of the darkened and brownish regions in bruise’s loquats, the combined spectral information and the color features method is proposed to accurately detect the storage time of mild bruise’s loquats in this study. In order to reduce economic losses, different methods are used to deal with the loquats at the corresponding bruise’s time. Loquats with four types of bruise’s time, including 6, 12, 24, and 36 h, are studied. Models with four types of characteristics, including spectral information, RGB features combined with spectral information, HSI features combined with spectral information, and mixed color features combined with spectral information (mixed-spectral), are established based on linear discriminant analysis (LDA), support vector machine (SVM), and least-squares support vector machine (LS-SVM). The investigated 400 independent samples with four bruise’s time conditions are utilized to assess the classification ability of the proposed methods. The results indicate that the Mixed-RBF-LS-SVM model has the lowest errors, and the accuracies of storage time of mild bruise’s loquats at 6, 12, 24, and 36 h are 100%, 92%, 92%, and 100%, respectively. The overall accuracy of the LS-SVM model based on mixed-spectral is 96%, and it demonstrates that the combined spectral information and color features method can be used to accurately detect the bruise’s time of loquats. Finally, the LS-SVM model based on mixed-spectral is optimized by UVE, SPA, CARS, and GA, respectively; it is found that the UVE-LS-SVM model based on mixed-spectral is the best, and the overall accuracy is 92%. It also lays a foundation for future studies about detecting the bruise’s time of fruits with a high-precision, rapid, and nondestructive measurement.
1. Introduction
Bruise is defined as damage to fruit tissue due to the external forces which cause physical changes in texture or chemical changes in color, smell, and taste [1]. Bruise means that the fruit is more likely to be decayed, which not only affects other intact fruits [2] but also causes a serious economic loss during storage and distribution. Therefore, it is necessary to distinguish bruised fruit from undamaged fruit to improve fruit quality and prevent food contamination. Loquat is native to China, which is an economical fruit for both medicine and food [3]. From ripening to final sale, loquat needs to go through a series of picking, storage, packaging, and transportation processes. Consequently, it is crucial to detect whether a loquat is bruised and when it is bruised to save storage and transportation costs.
In the process of loquat picking and postharvest handling, bruised loquats are identified through the operator’s naked eye that it is affected by personal habits, light intensity, and subjective psychological factors, resulting in misclassification that mild bruised loquat as normal loquat [4], so it is crucial to find a method by which the high-precision, rapid, and nondestructive detection of the bruised loquats can come true. In recent years, the spectroscopic techniques have been used in agriculture, including preharvest and postharvest product quality and safety detection and sorting [5]. Xing et al. [6] used a Zeiss spectrometer to detect apple’s fresh bruises in Vis/NIR regions (400–1700 nm); a classification accuracy of more than 95% was obtained for both the sound and freshly bruised spots on the selected apple cultivars. Jiang et al. [7] combined short-wave and long-wave Fourier-transform near-infrared spectroscopy (FT-NIRS) to detect and grade sweetness levels of intact peaches and nectarines, with a classification rate of 66.7% and 86.6%. These studies demonstrate that the spectroscopic techniques hold great promise for the nondestructive evaluation of bruise susceptibility because they are generally rapid and nondestructive or noninvasive, and more importantly, a large amount of information about the internal conditions of fruit is provided by them [8].
Hyperspectral imaging, which integrates conventional imaging and spectroscopy, is an emerging technique, and the spatial and spectral information of the object is provided by it. The spectrum of each spatial pixel contains the signatures of the sample substances, which are presented at the corresponding spots on the hyperspectral image [9]. Munera et al. [10] used hyperspectral imaging in the spectral region (450 nm–1040 nm) to detect skin defects of loquats, with a correct classification rate of 95.9%. Li et al. [9] used hyperspectral imaging in the NIR region (782 nm–1000 nm) to detect skin defects of bicolored peaches, with a correct classification rate of 96.6%. Although the techniques described above have shown promised detection results, the defect detection is still mostly limited to determining whether the fruit is sound.
Nevertheless, it is particularly difficult to distinguish fruit bruises in the first days after damage occurrence due to the fact that the softening and browning of the fruit are not displayed immediately [11]. Therefore, it is essential to find a rapid, nondestructive, and high-precision detection method for the bruising time of the fruit. To quickly identify the bruise’s time of fruit, different methods are used to deal with the fruit at the corresponding bruising time, which prevents bruised fruit from being decayed and affecting other intact fruits and reduces economic loss during storage and distribution. Nturambirwe et al. [12] used Ensemble Subspace Discriminant (ESD) to classify apple bruise’s time at 1 h, 6 h, 18 h, 48 h, and 72 h with an accuracy of 85%. Zhu et al. [13] used Extreme Learning Machine (ELM) to classify apple’s bruising time at 1 minute, 24 h, 48 h, 72 h, and 96 h with 92.9% classification accuracy. Li et al. [14] used hyperspectral imaging technology based on spectral information combined with the image grayscale value of a single wavelength point to classify bruise’s time of peach, and the classification accuracy was 96.67% for more than 24 h. In the above studies, most scholars use spectral information to determine whether the fruit is sound, but the spectral information is easily affected by external stray light resulting in losing some information. Meanwhile, the image features also contain information about the fruit, so it is necessary to combine spectral information and image features to improve detection accuracy.
Due to the enzymatic or chemical oxidation of phenolic compounds, damaged tissues become darkened and brownish in a few hours [15], and the color features can perform the conditions of the darkened and brownish region in loquats. Therefore, the combined spectral information and image color features method is proposed to achieve high-precision, rapid, and nondestructive detection of the bruise’s time of loquats in this paper.
2. Materials and Methods
2.1. Samples
In this study, the loquats are purchased from a local orchard (Panzhihua, China). To avoid the influence of other irrelevant factors, all loquats having similar size (major axis diameter is 60 mm and minor axis diameter is 40 mm) and weight (30 g) are selected for the experiment. Before the test, the samples are chosen to remove surface damage and deformities to ensure that the samples have no apparent defects, mechanical damage, and so on.
The surface bruising loquats are obtained by a free-fall collision device (as shown in Figure 1). To simulate the realistic loquat bruising, a metal ball with a diameter of 30 mm and a mass of 100 g is used to perform free-fall motion at 30 cm on the surface of the loquat, and the equatorial region of the loquat is impacted to simulate the realistic loquat bruising. When the switch is closed, the injured area of the loquat sample is located by the electromagnetic induction device with infrared light. When the button is opened, electromagnetic induction occurs at the end of the electromagnetic induction device, and the collision metal ball is fixed. Finally, the switch is closed, the magnetic field disappears, and the metal ball carries out free-fall motion to bruise loquats. The loquat samples for (a) sound and (b) bruised are shown in Figure 2.
(a)
(b)
2.2. Hyperspectral Imaging Acquisition System
The experimental image data are acquired by Gaia hyperspectral sorter. As shown in Figure 3, the hyperspectral imaging acquisition system consists of a computer, an imaging system, a light source system, and an electronically controlled displacement platform. The imaging system consists of a camera (Hamamatsu C8484-05G) and a spectrometer (ImSpector, V10E, Finland). The light source system consists of four 20 W halogen lamps (OSRAM, DECOSTAR51, MR16). The electronically controlled displacement platform consists of a carrier table and a stepper motor. All devices are installed in a dark box with 790 mm × 1024 mm × 1800 mm to eliminate the impact of stray light on the external environment. All of the acquired hyperspectral images with a size of 960 × 488 pixels, including 176 bands at 3.4nm intervals within the region of 397.5 nm ~ 1014 nm. All acquired spectral images are processed and analyzed by the Environment for Visualizing Images software program (ENVI 4.5, Research System Inc., Boulder, CO., USA) and the image processing toolbox of MATLAB 2021a (The MathWorks Inc., Natick, USA).
2.3. Reflectivity Correction
The hyperspectral system needs to be preheated for 30 minutes before data acquisition to eliminate the effect of baseline drift and avoid the errors caused by the image acquisition process. The parameters of the hyperspectral imaging system are adjusted by SpecView software; in order to acquire the sample image accurately, the camera exposure time is set as 6 ms, the displacement stage advance speed of displacement stage advance is set as 1 cm·s−1, and the displacement stage retreat time is set as 2.5 cm·s−1 to save the sample acquisition time. Only one loquat sample is collected in each test, the stepper motor is used to drive the displacement platform to move the sample, and the experimental sample image is acquired by the camera continuously scanning.
Due to the interference of dark current in the CCD camera and the uneven intensity distribution of the light source under each band, the hyperspectral images are needed to calibrate in black and white. After the imaging system parameters are set and before the formal experiment starts, the lens is pointed at the Teflon white plate, and it is scanned to acquire the all-white calibration image. After all, loquat hyperspectral image acquisition is completed, and the lens covering is examined to obtain the black plate image. The black and white calibration is shown as follows:where is the corrected spectral data, is the original spectral data, is the all-black spectral data, and is the all-white spectral data.
2.4. Least-Squares Support Vector Machine
The least-squares support vector machine (LS-SVM) is a kernel function learning machine that the principle of Structural Risk Minimization is followed [16]. It is an improvement of the support vector machine (SVM) that the inequality constraint of SVM is replaced by an equation constraint, the sum of squared errors is used as the loss function, and the quadratic programming problem is transformed into a system of linear equations problem to improve the speed and convergence accuracy of solving the problem.
Formula (2) is the discriminant equation of LS-SVM:where is the kernel function, is the input vector, is the Lagrangian operator, is the deviation, and N is the number of loquat samples.
2.5. Morphological Processing
Erosion, dilation, opening operation, and closing operation are the basis of morphological filtering, and they can be used to remove weak noises and reduce the effect of strong noises [17]. Both morphological opening operation and morphological closing operation can make the image homogeneities, and they can eliminate light and dark features, respectively. The calibration of morphological opening operation is shown as the following formula:
Similarly, the calibration of morphological closing operation is shown as the following formula:where , , , , , and are the input image, structuring element, morphological dilation, erosion, opening, and closing operations, respectively.
As the surface side of the loquat is curved and it cannot be fixed directly on the drive, a tray is needed to be placed on the drive to hold the loquat bringing the interference to the subsequent image segmentation, so the subsequent morphological operation is required to segment the loquat from the whole picture to eliminate the influence by the tray. Firstly, the image is binarized, and then the morphological erosion algorithm is performed to eliminate the impact of the tray. The erosion operation is like a minimum value filtering operation, a convolution kernel (or template, structure element) is used as the convolution when the edge of the tray is eliminated, and the morphological corrosion operation algorithm also erodes the loquat image, so the expansion algorithm is needed to recover it. The expansion operation algorithm is the dyadic operation of the corrosion operation algorithm, and it is a maximum filtering operation. In this experiment, the morphological corrosion operation algorithm and expansion operation algorithm are performed by the convolution using structural elements of flat discs with a radius of 5 pixels.
3. Results and Discussion
3.1. Spectra of All Samples
To eliminate the differences in reflectance caused by different shapes of loquat surfaces, a consistent spectral extraction area should be maintained. The spectral and image information of loquat samples are collected by ENVI4.5 software. As the spectral data of individual pixel points are not representative, the average spectral data of 200-pixel points are selected as spectra features by the Rectangular Region of Interest (ROIs). The spectral data are preprocessed by Multiplicative Scatter Correction (MSC) to eliminate the scattering effects of inhomogeneous particle distribution and particle size [18].
The reflectance spectra of standard loquat samples and different bruising times under mild damage are shown in Figure 4. The spectral waveforms of several grades of loquat are the same, and the peaks and troughs are also located at the same wavelength point, with different reflectance. At the same wavelength point in the region of 397.5 – 789.1 nm, with the increase of the bumping time, the reflectivity decreases, and the reflectance of the normal loquat sample is highest. Meanwhile, bruised tissues need a few hours to become darkened and brownish due to the enzymatic or chemical oxidation of phenolic compounds [1]. As the bruise’s time increases, the darkened and brownish regions of the bruise’s loquats and the reflectance of ROIs decrease However, in the region of 789.10–1014 nm, the reflectance of ROIs increases with the bruising time increase, and the reflectance of the normal loquat sample is lowest. That’s because that the bruised loquats are more likely to be invaded by the appearance, water loss and risk of bacterial and fungal contamination [19], and the enzymatic or chemical oxidation of phenolic compounds, resulting in the internal quality of loquat was changed with the increase of bruising time. Generally speaking, the characteristics of the bruised loquat are external features, and are detected in the visible region (400–780 nm). Meanwhile, the near infrared region (780–1000 nm) reflected the internal defects of bruised loquats, there are the difference of internal and external characteristics in bruised loquats. As show in Figure 4, the regions of 397.5–789.1 nm and 789.1–1014 nm reflected the external and internal defects in bruised loquats, respectively..
3.2. Extraction of Color Features
In this paper, the binarized template is used to remove the background to reduce noise for further extraction of color features [20]. The RGB features are the average gray values of the corresponding R, G, and B channels. Similarly, the HSI features are the average gray values of the corresponding H, S, and I channels. In real life, RGB features are superimposed in different degrees to produce a rich and wide range of colors, but it is difficult to describe the data accurately. However, the HSI color features are from the human visual system to describe color; the performances of hue, luminance, and saturation can be more clearly described by it.
Based on the above analysis, it is found that the model based on the RGB and HSI features combined with spectral information may have well classification results. Furthermore, the features of RGB and HSI combined with spectral information are used to establish the discriminant model.
3.3. Classification of LDA, SVM, and LS-SVM Models
Loquat samples of 6 h, 12 h, 24 h, and 36 h after bruising are divided into four groups, and they are labeled 1, 2, 3, and 4, with 100 samples in each group. The experimental samples are divided into modeling set and prediction set at the ratio of 3 : 1 by the Kennard-Stone method. As the grayscale values of the standard image are significant, the numerical values of spectral information are minor, so all of the features need to be normalized to reduce errors. Furthermore, the four qualitative analysis models are established by spectral information, RGB features combined with spectral information, HSI features combined with spectral information, and mixed features combined with spectral information, and the flowchart of loquat image information processing is shown in Figure 5.
Linear discriminant analysis (LDA) is one of the most common algorithms for supervised classification [21], and the basic idea of it is to transform the sample data into a new feature space by projection so that the distance between classes is maximized and the distance between samples within a class is minimized. LDA can be used to reduce the dimensionality of the feature space to a certain extent, and it also can be used to effectively extract classification information. In this study, LDA is used to evaluate the groups of storage time at 6 h, 12 h, 24 h, and 36 h. From all the classification results of the models of LDA, as shown in Table 1, it is found that the accuracy of the groups of storage time at 6 h and 36 h is more than the groups of storage time at 12 h and 24 h. The total accuracy is 76%, 81%, 80%, and 85% for storage time at 6 h, 12 h, 24 h, and 36 h, respectively. The results show that the accuracy of the spectral model can be increased by RGB and HSI color features, so the mixed-spectral features are determined as the optimal features.
Support vector machine (SVM) is a new machine learning method derived and developed on the basis of statistical theory, whose theory comes from the processing of data for classification. Numerous studies have shown that SVM can show certain advantages in solving small sample data and nonlinear data, such as its generalization ability and prediction ability [22]. In this study, SVM is used to evaluate the groups of storage time at 6 h, 12 h, 24 h, and 36 h. From all the classification results of the models of SVM, as shown in Table 1, it is found that the accuracy of storage time at 36 h is the highest. The total accuracy rate increases with the increase of characteristic variables, and the overall accuracy is 78%, 79%, 82%, and 87% for storage time at 6 h, 12 h, 24 h, and 36 h, respectively. The results show that RGB and HSI color features can effectively enhance the model discrimination ability.
The least-squares support vector machine (LS-SVM) is derived from the SVM algorithm, and it has the advantages of the SVM algorithm. The basic idea is to find the optimal classification hyperplane by mapping the input vector to a high-dimensional space through the kernel function. The inequality constraint is converted into solving linear equations to accelerate the solution speed [15]. In this study, LS-SVM is used to evaluate the groups of storage time at 6, 12, 24, and 36 h. From all the classification results of the models of LS-SVM, as shown in Table 1, it is found that the accuracy of the groups of storage time at 6 h and 36 h is more than the groups of storage time at 12 h and 24 h. The total accuracy rate increases with the increase of characteristic variables, and the overall accuracy is 79%, 95%, 89%, and 96% for storage time at 6, 12, 24, and 36 h, respectively. The results show that RGB and HSI color features can effectively enhance the model discrimination ability.
Based on the classification outcomes for all 16 combinations of 4 features and 4 bruising times by the above three methods (i.e., the LDA, SVM, and LS-SVM model) shown in Table 1, the optimal feature and model for the bruised recognition are determined.
3.4. LS-SVM Model Optimizations
Based on the classification results for all 4 features by the above three methods (i.e., the LDA, SVM, and LS-SVM model), it is found that the LS-SVM model based on mixed-spectral feature is the best for the bruised recognition. However, the model performance is evaluated by full-band data in the above analysis. In fact, this method is not suitable for use in rapid, automated, and in-line sorting in a large loquat production environment, so the model is necessary to be optimized. The variable filtering methods are used to reduce the dimensionality of spectral data, including uninformative variable elimination (UVE), successive projections algorithm (SPA), competitive adaptive reweighted sampling (CARS), and genetic algorithm (GA) [23]. In this paper, the LS-SVM model is optimized by UVE, SPA, CARS, and GA, respectively, the number of spectral variables is reduced from 176 to 38, 10, 30, and 4, respectively, and the results are shown in Figure 6.
As shown in Table 2, the mixed color features combined with the optimized variables of spectral information are used to be established, and it is found that the accuracy of all the optimized models is not as good as the Raw-Mixed-spectral-LS-SVM model. Compared with the number of spectral variables, it is found that the detection accuracy increases with the number of variables due to the fact that the features are lost in dimensionality reduction. Generally speaking, the spectral information reflects the characteristics of the sample, so the more characteristic bands after selection, the higher the accuracy of the model. However, sometimes there is redundancy between adjacent spectra in the spectral region. Using wavelength selection method to eliminate redundancy is helpful to improve the accuracy of prediction model [24]. Based on different data, the number of characteristic wavelengths obtained by using the same wavelength selection method is also different, so the performance of the established model is also different. In this paper, the UVE, SPA, CARS, and GA methods are used to optimize the model, respectively, the accuracy of the model optimized by UVE is the highest, and the overall accuracy of it is 92%. Although the accuracy of the optimized model is reduced, it saves a lot of detection time. Consequently, it also lays a foundation for future studies about detecting bruising time with a rapid and nondestructive measurement.
Four kinds of optimized models are compared, the results are shown in Figure 7, and it is found that the misclassification is mainly caused by classes (b) and (c). The UVE-LS-SVM model based on mixed-spectral can identify the class (a) and (d) samples perfectly. When it is used to distinguish class (b) samples, one sample is misclassified into class (a), four samples are misclassified into class (c), and one sample is misclassified into class (d). When it is used to distinguish class (c) samples, two samples are misclassified into class (b). Based on the above analysis results, the accuracy of class (b) should be increased in future studies.
(a)
(b)
(c)
(d)
4. Conclusions
This study demonstrates that the combined spectral information and color features method can be used to accurately detect the storage time of mild bruise’s loquat. In this paper, the models based on spectral information, RGB features combined with spectral information, HSI features combined with spectral information, and HSI features combining both spectral information and RGB features are established, respectively; it is found that the LS-SVM model based on mixed-spectral is the best, and the overall accuracy is 96%. Furthermore, the LS-SVM model based on mixed-spectral is optimized by UVE, SPA, CARS, and GA, respectively; it is found that the performance of UVE-LS-SVM is best, the overall accuracy is 92%, and the number of feature variables accounts for 21.59% of the total wavelength. This study provides a theoretical basis for subsequent qualitative discrimination of fruits by the combined hyperspectral spectral information and color features method. Consequently, it also lays a foundation for future studies about detecting bruising time with a rapid, accurate, and nondestructive measurement.
Data Availability
The data used to support the findings of this study are available from the author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
The authors gratefully acknowledge the financial support provided by the National Natural Science Foundation of China (no. 12103019) and the National Science and Technology Award Backup Project Cultivation Plan (no. 20192AEI91007).