Abstract
As the core component of agricultural robots, robotic grippers are widely used for plucking, picking, and harvesting fruits and vegetables. Secure grasping is a severe challenge in agricultural applications because of the variation in the shape and hardness of agricultural products during maturation, as well as their variety and delicacy. In this study, a fruit identification method utilizing an adaptive gripper with tactile sensing and machine learning algorithms is reported. An adaptive robotic gripper is designed and manufactured to perform adaptive grasping. A tactile sensing information acquisition circuit is built, and force and bending sensors are integrated into the robotic gripper to measure the contact force distribution on the contact surface and the deformation of the soft fingers. A robotic manipulator platform is developed to collect the tactile sensing data in the grasping process. The performance of the random forest (RF), k-nearest neighbor (KNN), support vector classification (SVC), naive Bayes (NB), linear discriminant analysis (LDA), and ridge regression (RR) classifiers in identifying and classifying five types of fruits using the adaptive gripper is evaluated and compared. The RF classifier achieves the highest accuracy of 98%, while the accuracies of the other classifiers vary from 74% to 97%. The experiment illustrates that efficient and accurate fruit identification can be realized with the adaptive gripper and machine learning classifiers, and that the proposed method can provide a reference for controlling the grasping force and planning the robotic motion in the plucking, picking, and harvesting of fruits and vegetables.
1. Introduction
Robots are replacing humans in performing primary motions such as grasping, carrying, and placing in dirty, chaotic, and dangerous working conditions [1]. The gripper is a core component of a robot. It serves as the end-effector of the mechanical arm and performs tactile information perception and haptic data collection [2, 3]. The gripper plays a crucial role in many manipulation tasks. In the field of agricultural production, the picking and sorting of agricultural products is a fundamental operation performed by robot manipulators [4–6]. Agricultural products are delicate and easily bruised, and they vary in shape, texture, and size [7]. To ensure grasping compliance, robotic grippers that can realize a human-like grasp are extremely important.
The gripper, which serves as the end-effector and performs actions in direct contact with agricultural products, is a vital component that allows the robotic device to interact with the external environment. To manipulate agricultural products that are delicate, irregular, and sensitive to damage in agricultural applications, the gripper should be flexible and adaptable [8]. At the same time, secure grasping requires soft contact, force control, and slip prevention while the object is being handled [1, 9]. Among the main tasks in agricultural applications, the harvesting of fruits and vegetables is one of the most time-consuming and labor-intensive tasks and suffers from low efficiency and limited competitiveness [10, 11]. Bac et al. and Lehnert et al. utilized a suction gripper and an oscillating blade in the end executor of an agricultural robotic harvester to reap sweet peppers in a cropping environment [12, 13]. Steady harvesting is ensured by the suction structure, while the oscillating blade manipulates the delicate peppers nondestructively with no direct contact. Eizicovits et al. modeled and analyzed two grippers (Fin-Ray and Lip-type). The two grippers were compared using two sweet pepper cultivars [14–16]. The importance of integrating perception analysis in the gripper design stage was illustrated in these studies. Zhao et al. designed and tested a dual-arm frame equipped with two 3-DOF manipulators and two different types of end-effectors [17]. The manipulators were used to pick tomatoes in a production environment without direct contact with the delicate fruits or damage to the agricultural products. Davidson et al. presented a preliminary design for a two-link planar manipulator, which incorporates the advantages of both the shake and catch method and selective robotic harvesting, to catch fresh apples [18]. Agricultural robots have been extensively applied in the plucking, picking, and harvesting of fruits and vegetables [19]. The performance of the end-effector directly affects the degree of success in such agricultural robot operations. To ensure secure and steady grasping, it is imperative to combine fruit classification, adaptive robotic grippers, and tactile sensing perception.
On account of the changing physical attributes of fruits during the growth process as well as the variety and delicacy of natural products in the harvesting environment, fruit classification is essential for adjusting the grasping strategy and motion planning in plucking tasks [1, 20]. Jiao et al. proposed a visual-tactile recognition method based on machine learning and multisensor information to recognize ordinary objects. The Kinect v2 was adopted for acquiring visual information, and bending and force sensors were embedded in the soft fingers to obtain tactile information [21]. Sun et al. presented two accurate object classification methods based on tactile sensing information that used the extreme learning method (ELM) and deep dynamical systems (DDSs). The ELM, which was based on visual clouds, was utilized for categorizing objects according to their shape and color [22]. In these studies, a high degree of recognition accuracy was achieved in a laboratory environment through the use of visual sensing information. Tang et al. reviewed the progress in the application of vision technology in harvesting robots [23]. Various visual algorithms have been widely applied in fruit reaping. However, because the lighting conditions are frequently unsatisfactory in agricultural applications, vision systems cannot accurately recognize the objects even when sophisticated vision algorithms are used in certain cropping environments [1, 24]. Integrated tactile sensors are key components of a robot hand system, which provide grippers with the ability to interact with their surroundings precisely, rapidly, and safely [25, 26]. The use of tactile sensing perception in robots can improve the safety of object manipulation and the efficiency of object identification. Many studies have been conducted on robotic grippers, which utilize tactile sensing information and machine learning algorithms, such as KNN [27, 28], RF [24, 29], and 3D convolutional neural networks (3D CNNs) [30]. These classification and recognition methods have been extensively studied and applied in various fields. However, accurate fruit recognition during fruit picking and harvesting remains a challenge. Zhang et al. presented a method utilizing array force information collected using only force sensors to recognize the hardness of fruits and vegetables [31]. They trained and tested two classifier models based on the KNN and support vector machine (SVM) algorithms with sample feature datasets obtained after dimensional reduction using principal component analysis (PCA). The grasping process can only be partially described by force sensors, and more comprehensive tactile perception is needed. In tactile-based methods, the deformation of the soft fingers, which depends on the size and shape of the fruits, can be perceived, and the grasping position can be detected using bending sensors [20]. The force magnitudes of the contact points between the gripper and fruit, which vary with the contact condition and stiffness of the target fruit, can be perceived by force sensors [25]. Accurate fruit classification and identification can be realized by sensing the applied force and the bending. In this paper, force distribution and bending deformation information are combined for fruit classification.
The fundamental purpose of this study is to classify fruits using a robotic gripper with machine learning algorithms and tactile sensing perception. The identification task can be performed by the learning algorithms based on the different variations of the geometric shapes and contact force of the soft fingers during the grasping process for different fruits. To successfully achieve this objective, the following subobjectives must be fulfilled: (1) design and manufacture of the adaptive gripper; (2) building of the tactile sensing information acquisition system and calibration of the tactile sensors; (3) development of the robotic manipulator platform and collection of tactile sensing data; and (4) performance of grasping fruit identification experiments.
2. Materials and Methods
2.1. Adaptive Robotic Manipulator
2.1.1. Structure Design and Working Principle
In this study, an adaptive robotic gripper was designed and shown in Figure 1. The robotic gripper acquires tactile sensing information during the grasping process. The gripper is composed of three mechanical modules, namely, a mechanical stent, a driver module, and a grasping module. The mechanical stent, which consists of a top transfer plate, a bottom plate, and frame plates, supports the entire frame and connects to the robotic device. The driver module provides power from a stepper motor to the gripper through a screw. The grasping module is formed by six sets of metal wires and silicone sheets. The silicone sheets directly contact the object being grasped. Linkage and presser bar mechanisms as well as sliding nuts are used in the transmission mechanism between the driver and the grasping modules.
(a)
(b)
(c)
During the grasping process, the target object is selected and the gripper is moved to above the object. The gripper is then dropped down slowly in a controlled manner, and the target object is grasped steadily. As the gripper closes, the motion of the linkage and presser bar mechanism are driven by the motor so that the metal wires and silicone sheets swing inward and make contact with the target object. The flexible and elastic silicone sheets allow the adaptive gripper to enclose the target object without damaging it.
2.1.2. Tactile Sensor Placement
Film sensors were attached to the soft silicone fingers using an adhesive. As shown in Figures 1(b) and 1(c), to measure the contact force between the fruit and the elastic silicone sheets as well as the deformation of the soft fingers during the grasping process, force sensors were adhered to the object contact surface of three selected fingers and bending sensors were placed on the other side of the fingers. The bending sensors do not directly contact the fruit. A bending sensor was integrated onto each finger, while two force sensors were evenly placed on each selected finger.
2.2. Integrated Tactile Sensors
2.2.1. Thin-Film Force Sensor
The force sensor is used in the fruit identification and classification tasks to perceive the contact force between the silicone fingers and the fruit and to discriminate its hardness and weight for fruit identification.
The force sensor (FlexiForce A201 1 lb) is an ultrathin and flexible printed circuit that can be easily integrated into the robotic gripper to monitor the contact force during the grasping and moving processes. With its paper-thin construction, flexibility, and high performance, the force sensor can serve as a data collector in the identification and classification tasks. The FlexiForce consists of an active sensing area and a banded thin-film circuit. The active sensing area is a 9.53 mm-diameter circle at the end of the sensor. The sensor can accurately measure contact forces between 0 N and 4.4 N using a strain effect technology, in which the application of a force to the active sensing area causes a change in the resistance of the sensing element in inverse proportion to the applied force. The sensor resistance is converted by the signal amplifier circuit into voltage values, which are then obtained by the microcontroller.
Before obtaining the tactile sensing data and identifying the various fruits, the force sensors were calibrated via measurement experiments. The sensors were calibrated in the linear range varying from 0 N to 4.1 N by applying increasing force at constant rates on the active sensing area and collecting the output ADC data using the tactile data acquisition circuit.
2.2.2. Flexible Bending Sensor
As the adaptive gripper performs the grasping operation, force sensing information, as well as morphological transformations and curvature changes, are generated on the surface of the silicone fingers. The bending information can play a significant role in shape discrimination during the manipulation of fruits and agricultural products with diverse varieties and sizes.
The bending sensor (Flex 2.2) placed on the gripper fingers is a flexible and durable sensor that can capture information from the soft fingers during grasping. Because of its small size, durability, and low cost, Flex 2.2 can be applied for the acquisition of tactile information in agricultural robotics. Flex 2.2 consists of a 2.2-inch-long strip-shaped sensing area and a conversion module. The conversion module is described in detail in Section 2.3. The Flex 2.2 is a resistance-based sensor, in which the resistance of the sensor changes with the curvature of the strip sensing area. The sensor is plugged into a signal conditioning circuit in order for the processor module to perceive the transformed bending information.
We define the flat state as 0° and the vertical state as 90°. The straight bending sensor was first placed on a horizontal plane, and its resistance was measured. The sensor was then bent gradually, and the resistance data were collected until the bending sensor reached the vertical state.
2.3. Tactile Sensing Information Acquisition
2.3.1. Data Acquisition Circuit
The tactile sensing data are used in the identification and classification tasks. The framework of the tactile sensing perception acquisition process is shown in Figure 2. In this section, the components and functions of the hardware are briefly introduced. An Arduino Mega 2560 is used as the microcontroller for the data collection circuit. Among its 70 I/O ports, 12 analog I/O ports are connected to the signal conditioning circuit to receive sensing information. The fruit identification system is illustrated in Figure 3.
Through the signal conditioning circuit, which utilizes an analog amplifier, the force sensors have the capability to accurately perceive the contact force between the fruit and the soft fingers. The signal conditioning circuit module is connected to the microcontroller analog I/O ports, as shown in Figure 3(a). The MCP 6004 is a low-cost and low-power instrumentation amplifier. The module is driven by a 5 V DC excitation voltage. We use the amplifier in the inverting operational amplifier configuration shown in the figure to produce an analog output based on the sensor resistance and an adjustable reference resistance. The output voltage is given bywhere represents the resistance of the force sensor, the output voltage, the power supply voltage, and the feedback resistance.
As a variable resistor with a resistance that varies with the bending degree, the bending sensor is used to measure the shape deformation of the fingers. The bending sensor measures the curvature change during the grasping process and transforms it into an analog signal through the conditioning circuit connected to the microcontroller. The conditioning circuit of the bending sensor and a schematic diagram of the amplifier circuit are shown in Figure 3(c). An MCP 6002 amplifier is used for bending information perception. The bending sensor acts as a variable resistor connected to the inverting amplifier circuit and determines the output voltage value that is collected by the analog I/O ports of the microcontroller during the grasping process.
2.3.2. Data Collection during Grasping Process
To achieve better performance in the tactile data collection and classification tasks, a uniform grasping process must be adopted for each operation. The object classification task is greatly influenced by the actuator position [29], as has been noted in many studies. To ensure that the actuator position did not affect the fruit identification results, a given grasping process was established for each operation, and the motor position and grasping motion necessary to eliminate the interference were determined. The threshold is selected near the maximum position of the actuator in order to ensure the stability of grasping and remove the influence caused by actuator position. At the same time, because of the adaptivity and softness of the robotic gripper, the grasping operation does not damage the agricultural products.
The grasping process is as follows: after a fruit has been selected, the adaptive gripper is moved so that it approaches the target fruit. The adaptive gripper is then driven to close and grasp the target fruit by the rotation of the motor. To verify the grasping stability, the target fruit is held and lifted for approximately 3 s. Finally, the target fruit is softly dropped. Particularly, in the grasping experiment, the posture of the fruit was random. Tactile sensing data are simultaneously obtained during the process by the acquisition circuit. The grasping process is illustrated in Figure 4.
(a)
(b)
(c)
(d)
In this study, five types of fruits, namely, apples, oranges, peaches, Korle pears, and tomatoes, which have diverse sizes, hardness, and maturities, were used as experimental samples, as shown in Figure 5. There were 50 of each type of fruit. The grasping, data collection, and identification tasks introduced earlier were performed on every fruit. Uniform operations were performed on the fruits, and the values from the force and bending sensors were recorded. In our experiment, the sampling frequency of the tactile information acquisition system was set to 15 Hz. A total of 5 × 50 sets of raw tactile data were recorded for each grasping process. Each dataset consisted of six sets of force and bending sensor reading series.
2.4. Classifier for Fruit Identification
The tactile sensing data collected during the grasping process are used to characterize the shape deformation and contact force distribution of the soft fingers, sense the differences in the sizes and stiffnesses of the objects, and evaluate the grasping quality [32, 33]. Various fruits can be identified based on multi-tactile perception using the proposed method and the adaptive gripper. In this study, a variety of machine learning algorithms were used to classify the fruits according to their tactile features extracted from grasping the fruits. The common classification algorithms RF, KNN, SVC, NB, LDA, and RR were employed.
The six classification algorithms are briefly introduced. RF is a nonlinear supervised classifier. The classifier contains multiple decision trees and is based on the bagging framework. RF provides high classification accuracy in distinguishing between multiple objects with different properties [29]. Compared to RF, KNN is one of the simplest classification algorithms. The extracted feature is distinguished for label assignment by mining the processed data to search for the k-nearest neighbors as measured by the distance metric defined in the KNN algorithm [27]. SVC is a classical linear classifier with a high generalization ability. The classifier is based on the principle of structural risk minimization. It maps the data to a high-dimensional eigenspace and segments the samples with a maximum edge hyperplane [34]. LDA is also a linear classifier. It projects the labeled data into a lower-dimensional space and searches for the vectors in the space that provide the best discrimination between the classes [35]. NB is based on Bayes’ theorem. Under the assumption of independent characteristic conditions, the NB classifier selects the category with the highest probability as the recognition result [36]. Finally, the RR classifier is a well-known linear regression method. Unlike ordinary least squares (OLS), it abandons unbiasedness and includes a regularization term [37].
The five types of fruits mentioned above, i.e., apples, oranges, peaches, Korle pears, and tomatoes, which have different sizes, maturities, and hardness, were used as experimental samples, and the same operations were performed on each fruit. For fruit identification, machine learning methods were used to classify the tactile features extracted from the grasping. There are six force and bending sensors acquiring tactile sensing information in the grasping experiment. First, original data collected by tactile sensors will be filtered to remove noise. Second, the tactile data sequence will be modified to a uniform length, which is 129 in our experiment. Third, the processed sequence will be joined to build a 11548 feature vector. Finally, 150 sets of tactile features will be operated as training sets, while other 100 sets will be test sets. 60% of the extracted tactile features were used as the training set, while the remaining 40% was used to evaluate the classification results.
3. Results and Discussion
3.1. Tactile Sensor Calibration
3.1.1. Contact Force at Grasping Point
It is necessary to calibrate the force curve and fit the force sensor so that accurate information on the contact surface can be obtained during the grasping process to characterize the grasping process, realize adaptive grasping, and distinguish the fruits. To calibrate the sensor, 110% of the maximum required test weight was exerted on the sensor to stabilize the sensor, maintained for approximately three seconds, and then removed. This process was repeated four or five times. The interface between the sensor and the grasping contact surface should be the same during conditioning, calibration, and actual testing. The conductance was converted to ADC by the conditioning circuit. Several sets of sensor data were measured to determine the relationship between the force and the ADC output.
The scatter diagram of the true ADC reading and the force in Figure 6(a) shows a linear trend. The linear equation is used to describe the ADC versus force relationship, and the coefficients and were determined from the collected data for the calibration. The linear fitting result was determined by ordinary least squares and is given in equation (2). The output, which represents the contact force between the soft fingers and the target fruit, was then fed into the classification models.where is the analog output of the force sensor conditioning circuit and is the contact force on the active sensing area. The contact force distribution on the contact surface can then be perceived using multiple force sensors via the ADC versus force relationship thus obtained.
(a)
(b)
3.1.2. Soft Finger Deformation and Bending Angle
The shape deformation of the soft fingers that occurred during grasping was captured and quantified by the bending sensors. The calibration of the Flex 2.2 bending sensor was performed using the signal conditioning circuit connected to the microcontroller unit (MCU) in the experiment. The variable resistance of the bending sensor was measured through the output voltage from the conditioning circuit, and the output data were collected on a personal computer through a serial port between the computer and the MCU. As shown by the scatter diagram of the measured calibration data shown in Figure 6(b), a linear relation between the resistance and the bending angle was demonstrated.
The resistance of the bending sensor changed linearly with the bending angle during the grasping process. The parameters of the linear equation were determined from the data collected in the calibration experiment. The linear relationship between the fitted sensor resistance and the bending angle is given bywhere is the resistance of the bending sensor and is the bending angle. The shapes of the various fruits can then be described by the sensor reading via the quantitative relationship between the true bending angle and resistance measurement thus obtained.
In addition, the test of other tactile sensors in our experiments was operated; similar linear characteristics were shown in Figure 6. The calibration experiment has verified the linear relationship between the sensors reading and actual tactile sensing.
3.2. Data Collection and Analysis
The performance of the robotic gripper was verified by using the adaptive manipulator equipped with tactile sensors to grasp various fruits. The tactile features were extracted from the tactile sensing information obtained using the uniform grasping process described in Figure 4 and used for fruit identification through tactile sensing perception and a machine learning algorithm. To verify the robustness and stability of the grasping, lifting, holding, and lowering grasping experiments were performed on the five types of selected fruits, which had different sizes, maturities, and hardness.
The original data obtained from the tactile sensors and acquisition circuit during the grasping process were represented as a set of numerical vectors in a high-dimensional time series. To remove interference from spike noise superimposed on the original tactile sensing data, a smoothing processing was performed using a 3rd-order median filer to process the collected tactile sensing data so that accurate fruit identification could be performed by the machine learning model. The curves of the obtained bending information and contact force series after the smoothing processing are shown in Figures 7 and 8, respectively. The tactile sensing data provided information for the adaptive grasping process and the tactile features for the classifiers.
(a)
(b)
(c)
(d)
(e)
(a)
(b)
(c)
(d)
(e)
Owing to the design of the transmission mechanism and robotic manipulator platform and the overshooting of the tactile sensors, the bending sensing curves obtained during the grasping process are nonmonotonic. The resistance curves of the bending sensors for various initial resistances and sensitivities are dispersed over the plots and have similar trends to those shown in Figure 7. Initially, the forward rotation of the stepper motor drove the metal wires to swing inwards to the maximum extent through the presser bar mechanisms. The contact with the target object and the force exerted on the fruit caused the shapes of the soft fingers to be deformed during the gripper closing process. This deformation resulted in the upward trend in the bending sensing curves because of the relationship between the sensor resistance and the bending angle. In the second step, the continued rotation of the motor caused the presser bar mechanisms to move up and the metal wires to swing back to arrive at the steady grasping state. In this step, the gripper was slightly loosened, and the deformation of the soft fingers was reduced. Because of the relationship between the sensor resistance and bending angle, the process of returning to the steady state caused a decrease in resistance in the bending sensing curves. In the third step, the fruit was held, lifted, and dropped during steady grasping. The sensor resistance returned naturally to a smaller stable resistance owing to overshoot attenuation. Finally, after maintaining a short steady grasp, the inverse process was performed, which was the opening process of the robotic gripper, and the corresponding resistance variation was observed. In particular, the resistance decrease before the first step was caused by advance contact between the fruit and the soft fingers because of the large size and irregular shape of the target fruit.
Similar to the resistance variation of the bending sensors, the force sensing curves were also affected by the transmission mechanism and grasping process. Owing to the faster adjustment speed and higher sensitivity of the force sensors, accurate contact force distributions were perceived in the grasping experiment. In addition, unlike the holding, lifting, and dropping processes of the bending curves, the force curves shown in Figure 8 exhibit fluctuations because of the mechanical properties of the soft materials and the undetected sliding of the contact surface.
The force distribution during adaptive grasping is well described by the distinction generated by different placements of force sensors. As shown in Figures 7 and 8, different tactile curves were obtained when different fruits were grasped. This distinction between the bending information and force distribution of different fruits during the grasping process inspires the use of machine learning methods to mine and classify the tactile data.
3.3. Fruit Identification Results and Discussion
In our method, machine learning algorithms are used for fruit identification and classification. Six machine learning models were employed to classify the fruits based on the obtained tactile sensing data. Model training and identification testing were performed. The classification accuracies and detailed parameters are listed in Table 1. For all the fruits in our experiment, tactile data from 30 of the acquired 50 grasps for each type of fruit were used for training, and the data from the remaining 20 grasps were used for validation. The accuracy of the classifiers varied from 74% to 98%. Almost all the experimental fruits were identified correctly by the RF classifier, which achieved an accuracy of 98%. The NB classifier achieved only 74% identification accuracy. Relatively higher accuracies were achieved by the other models.
The detailed identification results are presented in Table 2. The NB classifier failed in pear identification and had weak performance in the peach and tomato tasks, while the RF classifier misclassified only a peach and a tomato in all the identifications. Among the six classifiers, only the RF and LDA classifiers accurately identified all the experimental fruits with more than 90% accuracy, while the other classifiers obtained unsatisfactory results in certain fruit identification experiments. The SVC and RR classifiers accurately identified a large proportion of the experimental fruits, although several tomatoes and apples were misclassified by the two classifiers. Finally, unsatisfactory performance was obtained from the KNN classifier and NB classifier, with the latter having a success rate of only 35% in pear identification.
To display the classification effects more intuitively, the confusion matrices for the fruit classification by the various classifiers are presented in Figure 9. The diagonal elements correspond to correct identification results, while the nondiagonal elements correspond to misclassifications and indicate the identification probability and erroneous results. In most instances, the experimental fruits could be accurately identified by the trained classifiers. The RF and LDA classifiers performed better in the fruit identification task, while the confusion matrices of the KNN and NB classifiers indicate a more ununiform performance. Certain classifiers were confused by the similar sizes and hardness of apples, peaches, and tomatoes. RR, KNN, and NB successfully identified apples 75%, 85%, and 90% of the time, respectively. Most of the erroneous predictions occurred for peaches while a lower proportion was for oranges and pears. The result was attributed to similar sizes and irregular shape of apples and peaches. Meanwhile, some tomatoes were misrecognized as apples by the SVC classifier. A few tomatoes tactile features resembled the apples, which may be caused by random posture and similar sizes. The identification of pears by KNN and NB was unsatisfactory, with most of the pears being misrecognized as oranges. The large irregularity in the shape of the pear generated an irregular contact force distribution, which may have interfered with the judgment of the classifiers. In contrast, the regular size and shape of oranges led to their accurate identification by the classifiers. Unsuccessful identification may occur between fruits with similar sizes and hardness, or fruits with irregular shapes. In addition, the lack of tactile data and inadequate model training are also possible causes of the erroneous identification by certain classifiers.
(a)
(b)
(c)
(d)
(e)
(f)
The experimental results show that the RF and LDA classifiers are more effective for fruit identification using an adaptive robotic gripper with integrated sensors. The proposed tactile identification method can accurately distinguish between fruits of various sizes and diverse hardness and is useful for nondestructive operations in agricultural production through grasping force control and planned robotic motion.
4. Conclusions
In this study, a fruit identification method utilizing an adaptive gripper with tactile perception and machine learning classifiers was proposed. A motor-driven adaptive robotic gripper was designed for the adaptive grasping of fruits and agricultural products. A tactile data collection circuit was built, and tactile sensors were calibrated and placed on the soft fingers of the gripper. A robotic manipulator platform was developed, and the tactile sensing data were collected during the grasping process. The collected and processed tactile data were fed into the classification models to train and test the classifiers. Finally, to identify the type of fruit during the grasping process, six classifiers (RF, LDA, SVC, RR, KNN, and NB) were developed and evaluated using tactile sensing information obtained from the grasping process. Numerous fruit grasping and identification experiments were performed. The results demonstrate that the proposed method can effectively classify and accurately identify the fruits. The RF classifier achieved the best identification result with 98% accuracy, whereas the NB classifier achieved only 74% accuracy. Overall, imperfect identification results were obtained from the SVC, RR, KNN, and NB classifiers, and the apples, peaches, pears, and tomatoes were partially misclassified in these classifiers.
In agricultural production, targeted operations on delicate fruits and vegetables without damage to the products are currently a significant challenge. The identification and estimation of the attributes of agricultural products, such as their category, size, and hardness, can play a part in grasping planning in agricultural robots and greatly reduce the extent of damage to delicate agricultural products and economic losses in the production process. Our results demonstrate that the adaptive manipulators with tactile sensing perception can be used to identify the fruits and vegetables in plucking and sorting tasks and realize secure grasping of the object fruits with controlled grasping force. The identification performance can be improved in future works by using more accurate sensors, more ingenious manipulator designs, and more efficient sensor placements. The mechanical attributes of the fruits and vegetables can be described more comprehensively for recognition by tactile sensing perception and machine learning algorithms. In addition, the adaptive grasping based on tactile sensing perception and machine learning models should be able to accurately identify and classify various agricultural products with irregular shapes and diverse hardness.
Data Availability
The data will be available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest regarding the publication of this study.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (Project no. 31901415), the Natural Science Foundation of Jiangsu Province (Grant no. BK20180515), and the Fundamental Research Funds for Central Universities (Project no. KYYJ202110).