Abstract
Tactile sensors are one of the major devices that enable robotic systems to interact with the surrounding environment. This research aims to propose a mathematical model to describe the behavior of a tactile sensor based on experimental and statistical analyses and moreover to develop a versatile algorithm that can be applied to different tactile sensor arrays to enhance the limited resolution. With the proposed algorithm, the resolution can be increased up to twenty times if multiple measurements are available. To verify if the proposed algorithm can be used for tactile sensor arrays that are used in robotic system, a force sensing array (FSR) is adopted. The acquired two-dimensional measurements were processed by a resolution enhancement method (REM) to enhance the resolution, which can be used to improve the resolution for single image or multiple measurements. As a result, the resolution of the sensor is increased and it can be used as synthetic skin to identify accurate shapes of objects and applied forces.
1. Introduction
Recent studies have proposed various approaches to integrating multiple tactile sensors into a robotic system. The interaction between robotic systems and external environment has been facilitated by the implementation of various sensors, which can provide important data such as force, position, and shape. This work aims to present a tactile system that emulates a synthetic skin which can detect applied force and the geometric shape of the targeted objects. This system is based on the implementation of force sensing resistor (FSR) array because of its simple work principle and low cost. A single FSR is a resistor that varies its resistance depending on applied force. The measurements of the tactile sensor are first acquired by a supporting circuit and then connected to a data acquisition station. The major drawback of an FSR array is that its resolution and repeatability are both not satisfactory. The measurement of a particular force cannot always guarantee consistent resistance readout. With the inconsistent measurement, identification of geometric shapes cannot be accurate. As a result, in order to improve the resolution of such a tactile sensor array, an image processing method is necessary. In this study, a resolution enhancement method (REM) that can accurately improve measurement quality of a low resolution image is proposed.
The resolution enhancement is an interpolation process of calculating the values of intermediate pixels between the known pixels of certain low resolution measurement [1]. Such kind of methods is commonly divided into different categories, such as bilinear interpolation, nearest neighbor interpolation, bicubic interpolation [2], and adaptive cubic spline [3]. These techniques are highly used in image processing to enhance the image resolution. Many studies using these techniques to develop new algorithms have been reported. For instance, the existing Bicubic Convolution algorithm can convey low resolution images into sharper edges images. However, it also produces certain undesirable results when some textures details need to be preserved. In order to compensate for this shortcoming, Dengwen [4] proposed a new algorithm based on Bicubic Convolution (BC) Interpolation, which can produce sharp edges as well as texture details. Other than these methods, super-resolution algorithm [5–7] has been widely investigated in image processing field as well. One of the major advantages is that this method can greatly improve limited resolution images acquired by low cost imaging devices. It can generate high-quality images by integrating multiple low resolution and blurred images within the same targeted area. This technique has been applied to specific fields, such as satellite data, surveillance images, and medical imaging.
To obtain better image quality, Chu et al. [8] proposed an improved super-resolution algorithm using a gradient-based adaptive interpolation. This approach not only takes into account the distance between pixels but considers the local gradient of the low resolution image. The importance of the pixels is determined by the value of the gradient. To improve the quality for single low resolution image, Yang et al. in [9] proposed a method applied to a single image using sparse representation.
From their results, super-resolution algorithm has demonstrated an excellent performance compared with the conventional techniques for image processing. Some researchers also demonstrate that this method can be applied to enhance the resolution of two-dimensional data acquired by various instrumentation [10, 11]. Thus, this method can be a means to improve the measured quality of data acquisition devices with low resolution. In this research, a resolution enhancement method modified from super-resolution algorithm that applies to low cost tactile system to acquire force, position, weight, and shape of a targeted object is proposed. With the adopted FSR array, three major objectives need to be fulfilled. They are (1) identification of a statistical model of the tactile sensors; (2) development of supporting hardware and software interface that can be easily operated by users; and (3) implementation of a resolution enhancement method to improve the image quality acquired by the FSR.
The organization of the paper is as follows. In the second section, we will discuss the statistical model and the corresponding validation of an FSR sensor array. The third section discusses the theoretical background of the proposed resolution enhanced algorithm (REM) based on super-resolution method and the implementation procedures. Experimental results of both single measurement and multiple datasets are demonstrated in the fourth section. Summary and conclusions are given in the last section.
2. Modeling and Experimental Setup of FSR Array
A Force Sensing Resistor (FSR) is a resistive sensor that can be used as a tactile sensor in engineering applications. This type of sensor changes its internal resistance corresponding to the applied force. The relationship between applied force and output resistance is inversely proportional, which signifies that the resistance gradient decreases as applied force increases. Typically, an FSR device is mainly composed of two substrates of resistive polyester, which represent the top and the bottom circuit, and a spacer sandwiched between these two surfaces. Physically, material and geometrical specifications of the spacer and the layers determine the force necessary to make contact between the two substrates and hence to start varying the resistance.
FSR can be manufactured as a single sensor to measure applied force and also an integration of multiple FSR can be synthesized as an array of a fixed number of sensors to measure force distribution. For instance, Sensitronics [12], a provider of tactile devices, supplies two types of FSR arrays: Thru Mode and Shunt Mode. A Shunt Mode sensor array integrates 60 sensors accommodated in 10 rows by 6 columns. A Thru Mode sensor array includes 160 sensors arranged as 16 rows by 10 columns. In this paper, a Thru Mode array is adopted due to its greater number of sensors. The layout and circuit configuration of a Thru Mode array are shown in Figure 1.
2.1. Voltage Divider of FSR
In order to read the varying voltage generated by an FSR, an additional resistor () needs to be added to the voltage divider circuit. Figure 2 illustrates the basic configuration of the circuit. The output voltage of the circuit is With this equation, the variation of can be obtained by measuring the output voltage . To guarantee that the data acquisition device can always read the voltage variation according to the resistive change, an optimal value of needs to be determined. Based on the experimental data, the range of resistance of a single FSR varies from 5.7 kΩ to 0.36 kΩ corresponding to an applied force between 2 and 90 N. However, a Thru Mode FSR array shifts the range between 30.8 kΩ and 0.33 kΩ for the same force interval. By considering these values and 1, the resistance of individual FSR sensors can be determined. Extremely large or small can yield a small readable range of resistance.
2.2. Configuration of Experimental Setup
In order to efficiently acquire readings of individual sensors of the Thru Mode FSR array, a supporting circuit that converts force to voltage was fabricated. Figure 3 shows a diagram of the circuit, which is capable of reading all 160 sensors of the Thru Mode FSR array. The circuit integrates two multiplexers, 74HC4067E, which can control up to 18 channels separately. One multiplexer is in charge of selecting the row and the other one is in charge of selecting the column. The states of the multiplexers are controlled by four digital inputs. The purpose of the use of these devices is to simplify the 160 individual voltage divider circuits. The voltage outputs are designated by a data acquisition (DAQ) device. An add-on resistor of 10 kΩ is part of voltage divider and the activation voltage is set to be 5 V. The output voltage can be acquired by an analog input channel of the DAQ device. The analog outputs and digital inputs are processed by a MATLAB script.
The data acquisition device used is a NI-PCI-6221 DAQ card manufactured by National Instrument, which is controlled by the MATLAB script. The script is then in charge of converting acquired voltages to corresponding forces and displaying the image of the geometric shape of the distributed force on the surface of FSR array. The graphic-user-interface (GUI) of the script includes three main parts: a three-dimensional block display of force distribution of every sensor, a 16 × 10 table that demonstrates the numerical readings of applied forces, and a two-dimensional image that shows the shape of the object placed on the array. Due to the switching mechanism, the steady-state of each reading from individual sensor is ~0.1 sec. Thus, the required time to complete 160 readings of all sensors is approximately 18 seconds. The GUI and the displayed distribution of the measurement are shown in Figure 4.
2.3. Statistical Models in Different Regions
The major advantages of an FSR array include the following: (1) the supporting circuit is easy to implement and (2) the cost of an FSR array is low compared with other types of sensors. However, the accuracy of such kind of sensors is typically an issue. For instance, identical force with same magnitude may not always yield the same resistance. Such a phenomenon is extremely severe for a force between 0 and 20 N. Thus, precision of FSR sensors within this specific range needs to be improved hardware-wise.
To identify the relationship between applied force and measured output of an FSR sensor in a Thru Mode array, 17 forces with different magnitudes between 0 and 90 N were applied to a single sensor to establish the possible statistical model. An identical force was tested twelve times for the purpose of consistence. However, since the array has 160 sensors, it is not realistic to identify all the 160 relationships. Instead, 22 sensors in 6 different areas were tested. Figure 5 illustrates the diagram of the sensor array and the locations of the selected sensors. Depending on the location, the sensors behave differently. The difference of measurements at different location can be caused by the existing stress and possible deformation while being fabricated. In particular, if the FSR array is wrapped on a curvilinear surface, the resistance of sensors under zero loads in different regions needs to be identified to compensate for preexisting stress. After analyzing the experimental data, all the tested points were classified into six different groups. Groups A, B, C, and D are the sensors located along the borderline of the sensor array, and Groups E and F are the sensors located in the central area.
Figure 6 illustrates distributions of statistical data of groups in different regions. In this figure, the solid lines are the mean value of the measurements and the dash lines are the bands that cover 99% of the measurement (or mean value 3 standard deviation). It is clear that the relationships between the resistance and the applied force are consistent if the applied force is greater than 20 N. Based on the experimental results, the statistical models of all six regions are , , , , , and , where is the applied force and is the generated resistance of the FSR sensors in different regions. From the statistical result, Groups A and B have similar properties, which can be further grouped in the same category. Due to the preexisting stress of manufacturing, Groups C and D can only have similar properties when the applied force is greater than a certain value though they are both on the short sides of the border. The functions of different groups are used to calibrate the measurement.
2.4. Model Validation
To validate the statistical models, another set of experiments on another Thru Mode FSR array was performed. In the experiment, a single sensor was selected from each group and a theoretical value of resistance was predicted using the corresponding models. The predicted values were then compared with the output of the selected sensors in the corresponding areas. A range of forces from 20 to 90 N was applied by a force gauge and the resistance was measured. Table 1 lists the difference between modeled resistance and the physical measurement.
As shown in the table, the maximum error is 4% and the average error of all groups is only ~0.8%. The negative error means that the physical measurement is smaller than the modeled resistance. As a conclusion, the mathematical model can accurately predict the value of the resistance caused by the applied force.
3. Resolution Enhancement Method
3.1. Modified Super-Resolution Method
The basic idea of the proposed resolution enhancement method (REM) is that it reconstructs a high resolution image by processing several lower resolution measurements from the same object with slightly different viewpoints. In order to reconstruct a high resolution output from low resolution data, two constraints need to be modeled to solve this ill-posed problem [9]. Assume that the sampled data is , which is the blurred and downsampled measurement of the original object, and is the original object. Thus, the relationship between and is where is the blurring filter corresponding to a possible perturbation and is the downsampling operator. To apply the inverting procedure, a linear interpolation among samples is adopted for a fast transformation. The proposed method includes the following phases: registration, fusion, and deblurring. In order to convert the two-dimensional acquired data from lower resolution to a higher resolution output, an estimation algorithm is developed. Among different methods, the maximum likelihood approach presented by Bresler and Macovski [13] is one of the most intuitive ways to define the optimal reconstructed signal , which satisfies where is a weighted matrix that decreases the impact of the input noise. To obtain the maximum likelihood status of the estimated information, a gradient interpolation is used. The interpolation among four adjacent samples is processed as a block. Each block is transformed from a matrix to an matrix. Hence, if the low resolution data is an image of elements or pixels, the high resolution output becomes an image of dimension . The diagram in Figure 7 illustrates the gradient interpolation applied to a 2 × 2 block.
Depending on the availability of datasets, the proposed REM method can be applied to a single dataset or multiple datasets for better image quality. In case only one low resolution measurement can be available, the procedure that reconstructs a high resolution image is based on gradient interpolation. The gradient interpolation of the th block can be represented as where is the gradient transformation matrix. The size of the transformation matrix is , which is a linear iteration approximation matrix. The size of is , which includes the four corners of the targeted area. After the transformation, the matrix becomes an matrix. The value is determined. Once all blocks are processed, individual blocks can be combined and the synthesized output becomes A gradient interpolation method is adopted to approximate the slopes among points due to its fast calculation speed.
3.2. Translation and Integration of Multiple Measurements
However, due to perturbations and limited resolution of the acquisition device, a single sample is usually not enough to identify all the details in most cases, which means some information can be missed. By taking multiple measurements, one of the samples is possibly to contain some details that other samples do not have. Thus, the missing information can be possibly collected and revealed if multiple measurements are acquired. In other words, the quality of the output image can be improved by applying the proposed REM method to multiple sets of measurements.
In order to acquire more details, multiple measurements with minor translations and rotations in either the acquisition device or the targeted object can be helpful. With the tiny movements, small details can be revealed. The main idea is that the acquired data of the targeted object always has a slight difference in every measurement. By collecting these tiny differences from various sampled datasets, unclear detail of existing parts will be concentrated. Hence, it is important to develop a method to register the position of the individual sampled datasets to a common reference map.
To find a common reference, many pattern recognition techniques [14, 15] were tested. However, with the limited resolution of acquired measurement, it is difficult to identify multiple objects that can align multiple images correctly on the same map. Thus, a contrast map was adopted to identify the area occupied by the targeted object in individual measurement. To build a contrast map, all measured samples were converted into a black-and-white image, which is determined by the difference among adjacent samples. If the differences are larger than a preset threshold, the specific locations are registered. Once the contrast information is collected, the original image can be divided into clusters. All clusters with the targeted object are identified on each sample data. Thus, a common reference map can be applied to bring all the areas to the same coordinates. With all targeted objects aligned to a common coordinate, the proposed REM can be applied to individual measurements [11]. The proposed datasets are then combined to obtain a single image that includes all the details from every image. The simplest combination method is to take the average of every pixel. That is, where is the number of total measurements and is the th processed dataset. Deblurring technique using a Wiener filter [16] can be adopted to remove the blurred effect caused by the operation of low-pass filter.
3.3. Resolution Enhancement of a Sensor Array
The adopted tactile sensor array is a Thru Mode FSR array provided by Sensitronics, LLC. This array is composed of 160 sensors, that is, a matrix. While measuring, the object is placed on the surface of the Thru Mode FSR array. The measurements of individual sensors are then collected by a DAQ station. However, due to the low resolution of the sensor array, the geometric shape of the object can barely be identified even after all the readings are taken. Due to the low resolution and repeatability of tactile devices, a resolution enhancement method is required. The procedure of data collection and data processing follows the steps discussed.
It is also required to have a metric that estimates the quantitative improvement of the proposed technique. However, it can be difficult to determine the improvement of the processed outcome without a reference. To quantize the quality of approximated outcomes, PSNR (peak signal-to-noise ratio) can be used to evaluate the distortion after approximation. The metric of PSNR between original measurement and processed measurement is defined as where is the size of the measurement. The luminance is the largest possible value of the signal. In general, the larger PSNR value implies that the reconstructed image is a better approximation of the original image. To evaluate the output quality of the proposed iteration method, it is recommended that the value of PSNR should not be lower than 25 dB—if the two datasets are converted to 8-bit grayscale images. The relative difference between two PSNR values, for two different reconstructed images, gives the metrics of the reconstruction or compression quality.
In this section, a measurement of two sets of data measuring the same sample with different resolutions was used to evaluate the effectiveness of the proposed technique. The measurement of Figure 8 is a roughness analysis data acquired using NanoScope manufactured by Digital Instruments. The original scan size is 7.593 μm with a resolution of 512 × 512 samples. To evaluate the proposed method with different sizes of approximation patch, the data was cropped to 480 × 480 samples. The image was downsampled to different sizes and then restored to the original size accordingly. The selected patch sizes are 2, 3, 4, 5, 6, 8, and 10. Based on the values of PSNR, the quality of restored image can be determined. From the results shown in Figure 8, the PSNR drops from 31.5 dB to 28.7 dB, which is ~3 dB when the patch size is 5. The quality can still be acceptable if the patch size is 10 since the PSNR is still greater than the threshold of 25 dB.
4. Experimental Results
4.1. Processing of a Single Set of Data
To verify that REM can be used in all kind of scenarios, several objects with different shapes and sizes are used. In the experiments, the object is placed on the surface of a Thru Mode FSR array with an 86 N force applied. The force was calibrated by a force gauge (FGE-20X, manufactured by Shimpo Instruments). The individual readings of all sensors are then acquired by the DAQ station and saved as a raw image. Based on the adopted FSR array, the resolution of the image is 16 × 10 pixels. This image is then processed by the proposed REM. To further improve the image quality, multiple images are used to compensate the missing parts of a single acquired image.
The object used is an annulus shown as Figure 9(a). Figure 9(b) demonstrates the raw data of the measurement. The processed results are shown in Figure 9(c) with . It is clear that the geometric shape of the annulus can be outlined even with only a single measurement. The image can be further enhanced by a deblurring algorithm.
(a)
(b)
(c)
4.2. Processing of Multiple Sets of Data
The results after processing and deblurring a single image are usually good enough to identify the shape of the targeted object. However, a single image can sometimes miss some detail due to two important reasons: the low repeatability of the tactile sensor and the physical malformations of the analyzed objects. In this section, multiple measurements of the same objects used in the previous section are processed; it is anticipated to generate a more accurate result.
In each case, twelve measurements were collected from individual objects. The object was placed at different positions on the surface of the adopted FSR array while the data was collected. The patch size used for multiple sets of data is 3 × 3. Figure 10 illustrates the twelve measurements of an “L” shape object. Once the twelve measurements of a specific object were collected, all the measured outcomes need to be shifted to a common reference individually.
To simplify the calculation, the area enclosing the object is considered to be within an square matrix. Each area has its unique associated coordinate () at four corners in its original measurement. This targeted area can be found by applying a predetermined threshold value to individual images and locating the boundary points of the four corners. As a result, the maximum and minimum points in - and -axis of individual images can be identified to enclose the object, which determine location of targeted area. Once the targeted areas are identified for individual measurements, REM is applied to the areas for resolution enhancement. After REM has been applied, centroids of the enhanced images are calculated. It is assumed that the centroids of the same object are located at the same position. Thus, by knowing the locations of centroids, all the targeted areas from individual measurements can be brought to a common coordinate.
The individual enhanced results are then merged to generate a final result that contains all the information processed. After translation and averaging, the processed and deblurred images are shown in Figure 11(a). Notice that the translation actually reduces the image size of processed result since all individual measurements are shifted to a common reference coordinate. Nevertheless, compared with the processed result from a single measurement as shown in Figure 11(b), it is clear that the result from multiple measurements actually reveals missing detail in the processed result of a single measurement even if the resolutions are identical. The improvement of quality can be evaluated by the metric of PSNR. The original sensor array is first gridded into a 30 × 48 map with the resolution of the desired patch. By overlapping the geometric shape on the map, all occupied cells are marked as 255 (gray scale), while others are marked as 0. Then the PSNRs of the two outcomes can be evaluated. With this method, the processed result from multiple measurements has higher PSNR. However, the PSNR can only be used to evaluate the similarity of geometric shapes. It cannot be used to evaluate the force distribution since the data of force measurement with high resolution is not available. Figures 12(a) and 12(b) demonstrate the processed results from multiple measurements and a single measurement of another “T” shape object. It is also clear that the results from multiple measurements can show better detail even if the resolution of the two processed data is identical.
5. Conclusion
The goal of this research was to propose an integrated tactile system that can be used as a synthetic skin, which can provide data of force distribution, weight, location, and geometric shape while contacting an object. The adopted Thru Mode FSR array has the advantages of low cost and convenient deployment. A mathematical approximation of the sensor array has also been identified. A supporting circuit and DAQ system were set up as well. Though the resolution is limited, the proposed REM method can greatly enhance the resolution. In particular, processing with multiple measurement can greatly improve the resolution of the raw data. The main contribution of this study was the implementation of the integrated system which provides a skin-like platform. In the future, the proposed algorithm will be applied to different platforms for other applications, such as AFM images and scanning of 3D images.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
The authors gratefully acknowledge NASA West Virginia Space Grant Consortium for the financial support of this project.