Abstract
In order to achieve robots' working around humans, safe contacts against objects, humans, and environments with broad area of their body should be allowed. Furthermore, it is desirable to actively use those contacts for achieving tasks. Considering that, many practical applications will be realized by whole-body close interaction of many contacts with others. Therefore, robots are strongly expected to achieve whole-body interaction behavior with objects around them. Recently, it becomes possible to construct whole-body tactile sensor network by the advancement of research for tactile sensing system. Using such tactile sensors, some research groups have developed robots with whole-body tactile sensing exterior. However, their basic strategy is making a distributed 1-axis tactile sensor network covered with soft thin material. Those are not sufficient for achieving close interaction and detecting complicated contact changes. Therefore, we propose “Soft Sensor Flesh.” Basic idea of “Soft Sensor Flesh” is constructing robots' exterior with soft and thick foam with many sensor elements including multiaxis tactile sensors. In this paper, a constructing method for the robot systems with such soft sensor flesh is argued. Also, we develop some prototypes of soft sensor flesh and verify the feasibility of the proposed idea by actual behavior experiments.
1. Introduction
In order to achieve robots working around humans, it is indispensable for such robots to have ability to contact safely against objects, humans, and environments with broad area of their body.
Many practical applications, such as nursing care or carrying a number of stuffs using whole upper body, include whole-body close interaction with humans or objects. Robots around humans should utilize soft contacts actively as humans or other animals do.
Historically, robotics research for interacting with objects or humans started from strict modeling of the contacts by limiting the contact points only at their end effectors. Of course, this methodology achieve a certain result. But, it is difficult to achieves whole-body interaction with objects or humans like other animals or humans by an extension of this methodology. As the development of tactile sensing system proceeds [1, 2], it becomes possible to construct whole-body tactile sensor network for robots. Using such tactile sensors, some research groups have developed robots with whole-body tactile sensing exterior [3–8]. Although there are some differences on how to achieve distributed tactile sensor exterior among them, their basic strategy is that they distribute many small 1-axis force/pressure sensors which detect normal direction force, and they are covered with thin soft material, which can be said as “skin.” However, various types of contacts, not limited to just a push, can be occurred during whole-body close interaction. For detecting such complicated contacts, multiaxis force or deformation sensing of the exterior is necessary. From that point of view, Iwata et al. proposed a whole-body tactile sensor cover which detects both position and multiaxis force vector applied on the sensor cover [9]. However, the module of their sensor cover which can detect both force position and vector is relatively large, and local multiaxis deformation of the soft exterior cannot be detected by their method. Also, for constructing soft tactile sensor exterior with detecting ability for local multiaxis deformation, sensor element itself should be soft and small. Although there are some studies about multiaxis tactile sensors embedded inside soft materials [10–12], they are in the developmental stage and it is difficult to use them for constructing whole-body tactile sensor exterior.
In this paper, we propose “Soft Sensor Flesh,” a new construction method for whole-body tactile sensor system. Human body have thick flesh structure embedded with many sensor elements in different depths, not the thin skin with distributed sensors. Basic idea of “Soft Sensor Flesh” resembles such human structure, that is, constructing robots’ exterior with soft and thick foam, inside which many tactile sensor elements including multiaxis deformation detectable sensors are embedded. For achieving this goal, not only the combination of existing sensors, but also new soft tactile sensors are necessary. Therefore, we also propose some soft tactile sensors as core functionality of “Soft Sensor Flesh”.
In the following sections, a constructing method for the robot systems with such soft sensor flesh is argued from both hardware and software aspects. Also, we develop some prototypes and verify the feasibility of the proposed ideas by actual behavior experiments. This paper is organized as follows: requirements for achieving soft sensor flesh is argued in Section 2. Sections 3 and 4 are basics of the soft sensor flesh. In Section 3, basic structure and construction method for soft sensor flesh is proposed, and the experimental results from the first generation of our soft sensor flesh is discussed. In Section 4, computer simulation model for soft sensor flesh is described. Next, Sections 5 and 6 propose improved soft sensor flesh: construction method for soft sensor flesh with integral molded sensors is proposed in Section 5, and new soft multiaxis deformation sensor is proposed in Section 6. Then, Sections 7 and 8 argue other possibilities for soft sensor flesh. In Section 7, soft sensor flesh for self-protective behaviors is argued. In Section 8, additional soft tactile sensors which compensate the contact sensing ability of the basic soft sensor flesh is proposed. Lastly, this paper is concluded in Section 9.
2. Close Interaction Behavior by Robots with Soft Sensor Flesh
By having soft contacts with humans or objects and detecting those contact states at the same time without any limitation of touched place, robots can conduct close interaction behavior safely. Humans and animals can propagate force effectively to other objects with the broad region of their body by closely interacting with those objects. Implementing such close interaction behavior for robots opens a gate to develop many practical applications: nursing care robots which hold up patients, robots carrying large stuffs during move, or shopping assistant robots holding some packages under its arm, and so forth. On the other hand, soft physical interaction can make a great influence on human’s mind. For constructing good communication, robots are expected to give humans more cuddles. Studying such robots is one way to achieve partner robots necessary in aging society with fewer children, such as baby sitting robots or elderly care robots.
In order to construct robots which can achieve such close interaction behavior, at least following requirements should be met.(i)Whole-body region is covered by soft exterior like humans or animals for being touched anywhere on the body.(ii)Tactile sensors detecting various contact states on real time are distributed all over the body.(iii)Multiaxis force/deformation sensing is done for detecting contact state changes caused by the change of applied force even when the touched places do not change so much during close contact states.
Considering those points, hardware and software system of the “Soft Sensor Flesh” is constructed. In the following sections, basic idea and the application of the robots with soft sensor flesh is argued. Firstly, basic idea of “Soft Sensor Flesh” and its construction method is proposed in Section 3.
3. Basic Structure of “Soft Sensor Flesh”
The basic idea of “Soft Sensor Flesh” is shown in Figure 1(a). Mechanical frame of the robot is covered with soft thick material, and multiaxis force/deformation sensor is embedded deeply inside the exterior. Also, other various sensors are molded and placed in the middle depth of the exterior. Figure 1(b) shows the schematic of our first prototype of the soft sensor flesh. Here, commercially available 3-axis force/torque sensor is fixed to the mechanical frame of the robot, and the externally applied force including shear component is propagated through thick layer of urethane foam. Selection of the soft thick material is one important point for constructing soft sensor flesh. The material of the exterior should be easily customizable for studying many variations of the construction method of soft sensor flesh. Also, it should be light enough for covering whole-body region of the robot. Considering those requirements, we have tested many materials (Figure 2). In Figure 2, blue boxes indicate conclusive methods. Green boxes indicate methods which we have tested ever. The gray boxes indicate methods which we have not tested. Lastly, the white boxes indicate methods which we have determined that test production is impossible.
(a)
(b)
Among possible soft materials shown in Figure 2 ((a) Soft Materials), we selected foam plastic. Compositive materials used for animatronic works such as rubber (exterior) and soft foam (interior) were ignored in this research, because they require technical manual skill for creating robot exterior by ourselves. Also, we have made prototypes of costume for robots with cotton and stretchable fabric before. But, these choices are rejected because of the difficulty in making a precise copy of 3D CAD model. Specific gravity of silicone rubber (Shin-Etsu Chemical Co., Ltd.) and urethane gel (Exseal Corporation) is higher (about 1.0) than that of foam plastic (about 0.05–0.1). Degrees of heat conductivity of silicone rubber and urethane gel are higher (about 0.2–0.4 W/mK) than that of foam plastic (about 0.02–0.07 W/mK). In order to construct whole-body exterior with enough thickness to embed various types of tactile sensors inside, both weight and heat problem should be considered. Here, we place much value on reducing total weight. Thus, we decided to use soft foam plastic with concern to heat problem. Next, among soft foam plastics ((b) soft foam plastics), representative examples of soft foam plastics are made of polyurethane, polyethylene, and rubber. Polyethylene foam and rubber foam are produced by kneading base compound and bloating agent and then heating them, which are relatively difficult to produce experimentally. Some kinds of soft polyurethane foam can be molded by simply mixing base compound and curing agent at room temperature. Also cure time is not long (several minutes). Therefore we decided to use soft polyurethane foam. For processing methods ((c) processing methods), molding method is adopted. In order to give form to soft polyurethane foam, there are three methods. First method is cutting, grinding, and press work of slab foam, second method is molding, and third method is spray foaming. Spray foaming does not require time-consuming measuring and mixing process. However, commercially available spray foaming polyurethane foam is not suitable for our application in hardness and shape forming. Soft polyurethane foam deteriorates due to ultraviolet rays, mechanical friction, and heat. Deteriorated foam turns yellow then crumbs. This crumbing problem was observed more often on slab foam than on molded foam, which has skin layer on the surface. In consequence, we employed molding method. Next, finishing surface of the exterior ((d) surface finishing) should be considered for avoiding significant deterioration of the surface. In-mold coating and film laminating have fine texture and durability capacity. But, these methods require precise temperature and pressure management and expensive mold. Thus, we employed painting method, especially liquid rubber for coating, taking care of surface appearance and odor. Both molded foam and paint film contract with curing. Therefore it is necessary to design molds larger in advance. We measured its shrink ratio (about 1.5%) experimentally. Lastly, molded foam is covered by stretchable cloth for achieving comfortable contacts against humans.
In Figure 3, our first prototype of a humanoid robot with soft sensor flesh is shown. This robot is named “macra” [13], and “macra” is a small humanoid which has soft urethane foam exterior and embedded 3-axis force/torque sensors. The height is about 700 mm, and the weight is about 7.5 kg. Its internal mechanical frame has 22 DOFs. Some parts of the internal frame and circuit boards for driving actuators are built using HOAP-1 manufactured by Fujitsu Automation Ltd. [14]. As embedded tactile sensors, 49 small 3-axis force/torque sensors (PFS, manufactured by Nitta corporation) are used. Since they are embedded in the soft urethane foam and can detect three-dimensional force directions, deformation forces applied to the soft exterior can be calculated by the interpolation of the distributed force sensors. Therefore whole-body contacts of macra can be detected as a bundle of the flow of the applied forces. Figure 3(c) shows some examples of macra’s sensing abilities. The sensor output vectors are illustrated as red arrows from corresponding sensors in the robot’s geometrical model, and 3D force directions can be seen.
(a)
(b)
(c)
In Figure 4, outputs of one 3-axis force/torque sensor during close interaction behavior are shown. On Figure 4(a) the output of a 3-axis force/torque sensor which is located on front body of the robot is shown. It shows that there is enough difference from each other to be detected separately while a human touches in a various manner. Figure 4(b) shows an application case. Softness of the exterior enables the robot to hold a bundle of objects. Soft exterior deforms and fits the objects. If a person tries to take the objects away from the robot, output of the sensor at right forearm in - direction changes. In this case, the person pulled the bundle of objects up and down (Figure 4(b)).
(a)
(b)
In Figures 5 and 6, tactile sensor output changes during close interaction behavior are drawn on a geometrical model of the robot. In the geometrical model, 3D force vector is expressed as white arrows. In Figure 5, it is confirmed that a human applies not only pushing force, but also shear force to the robot’s torso. Also, Figure 6 shows sensor output changes during self-collision.
In this section, basic structure of “Soft Sensor Flesh” is proposed and the idea is confirmed by the developed first prototype. In the next section, how to emulate the tactile sensor outputs of this soft sensor flesh in computer simulation is argued.
4. Computer Simulation Model of Soft Sensor Flesh
In order to test various behaviors, acquiring behaviors by evolutionary methods such as GA (genetic algorithm) is one good way. In this section, implementation of a computer simulation model of soft sensor flesh for acquiring whole-body contact behavior using GA is described [15]. Here, robot programming environment EusDyna [16], which integrates robot motion programming environment and dynamics simulator, is used. In Figure 7, approximate model for macra in EusDyna is shown. In the simplified link model at Figure 7(a), yellow boxes attached to the links indicate 3-axis force/torque sensors. Red arrow and orange arrows show the direction of gravity and contact forces between soles and ground. Figure 7(b) shows a model with flesh. Each flesh models are approximately constructed by combining spheres, rectangular solids, and capsules (cylindrical structures of half spheres at both ends) and attached to corresponding links. The weights of those flesh models are decided by the actual measurement of the macra’s flesh parts. To express flesh viscosity and elasticity, those flesh models are allowed to intersect with each other to some extent.
(a)
(b)
Since all sensing points are embedded in the soft flesh, it is not enough to simulate tactile sensing ability with dynamics models alone. In order to emulate sensor outputs in a real environment, flesh and sensor properties actually used in macra should be considered in the simulation model. In Figure 8, tactile sensor used in macra is shown. As shown in Figure 8(a), 3-axis force/torque sensors (PFS) are fixed to the internal frame. A sensing plate is used for broadening sensing area. Figure 8(b) shows the schematic of the sensor. This 3-axis force/torque sensor detects , (torques against horizontal axes), and (perpendicular force against the plane at which the sensor is fixed). Here, only is considered for modeling this sensor to simplify the model. From the dynamics simulator, contact forces at fixed contact points on the surface of soft exterior are given. To emulate actual sensor outputs, those contact forces should be converted to the outputs of the distributed 3-axis force sensors. Sensor and flesh properties are modeled for conversion.
(a)
(b)
As a sensor property, force-voltage characteristics are modeled as a linear function. When they are expressed as in (1), whereas kgf/V is gain, bs kgf is offset, and ns kgf is noise, the first two parameters are decided by actually measuring the macra’s force sensor with a 1-axis digital force gauge (IMADA DPX-50T). From temporal data recorded on input force and output voltage, parameters that give the least square errors are chosen. Also, Gaussian random number is introduced as a noise model since the sensor noise is not negligible. The amount of noise ns is calculated as a standard variation of 125 samples of sensor output voltage under no load. Because of individual sensor differences, parameters are calculated for each of the 49 sensors used for macra.
As a flesh property, force propagation of the soft flesh is modeled. Especially, spatial decay is examined here. In order to collect sufficient data, we use a test piece for soft sensor flesh, that has uniform thickness and 1 embedded 3-axis force sensor. The proportional relationship between load and transferred force is assumed, where is the -axis component of force applied by vertical pressing on the surface, and is the -axis component of force detected by the 3-axis force sensor (see schematic model in Figure 9(a)). The proportional coefficient is expressed as force transmission rate . The force transmission rate decays as the distance expands between the contact point of the test piece and the center of 3-axis force sensor. In this case, is modeled by a 2-dimensional Gaussian of horizontal distance and vertical distance . To collect data, three types of flesh with different thicknesses ( is 2, 30, and 60 mm) are prepared, and is calculated at different horizontal distances. Figure 9(b) shows the distribution of the collected data, where the horizontal axis is horizontal distance mm, and the vertical axis is force transmission rate kgf/kgf. In the graph, each line expresses the fitted 2-dimensional Gaussian model from collected data. As a result, (2) is calculated as the force transmission decay model.
(a)
(b)
Using this simulation model for soft sensor flesh, we have developed a system for acquiring rolling-over motion based on tactile sensor information. For acquiring sensor-based behavior that closely interacts with the environment, behavior system with tactile sensor information feed-back for the current motion is necessary. An artificial neural network connecting sensor values and joint angle commands achieves such behavior system, and optimized parameters of the neural network for rolling-over motion are acquired using a GA. In behavior acquisition process, evaluation of each trial is repeated in the above simulation environment until fitness improving is converged. The acquired neural network is tested by using macra in a real environment.
In Figure 10, schematic image of the constructed system is shown. While behavior is being acquired, calculation is done in the simulation environment. After behavior is acquired, the actual robot body receives joint angle commands from “Motor model” and sends sensor information to “Sensor model.” From the real or virtual robot’s body, three types of sensor information are sent to “Sensor model” in Figure 10. In “Sensor model”, each sensor value is normalized from 0 to 1. As a result, (normalized tactile vector: 49 dimensions, force toward normal direction, that is, -axis for the sensor), (normalized angle vector: 22 dimensions), and (normalized gravity direction vector: 3 dimensions) are obtained. Normalized signals are aligned and a 74-dimensional sensor vector is input to the neural network controller. Output of the neural network controller is a 22-dimensional vector , relative changes against the current angle vector. In “Motor model”, these changes (values from 0 to 1) are converted to an adequate range from −5 to 5, and added to current angle vector . The updated angle vector is then sent to the robot’s body as a joint angle command. Thus, the nervous system having sensory feedbacks to joint angle commands is constructed. Basically, for correctly feeding back sensor inputs, update cycle for the robot’s body system should be shorter than that for the nervous system. In this study, update cycle for the simulation world is 25 ms, and that for the nervous system is 50 ms.
In the behavior acquisition phase, a real-coded genetic algorithm is used to search for optimized parameters. However, there is a problem that behavior acquisition phase takes too long because each gene evaluation takes some seconds to minutes in the dynamics simulation environment, and total evaluation time becomes some days. In order to reduce behavior acquisition time, parallel GA system conducted in the PC cluster environment is used here.
During acquisition phase, rolling-over motion is evaluated as follows (see (3)). In the equation, is the score for the th trail during times trails in one evaluation. is -axis components of the direction of gravity at the torso link (the front face of the torso is positive at the -axis of torso link coordinates). is the gravity acceleration. Therefore, the score is higher when macra lies on its stomach as soon as possible as a result of the rolling-over motion after starting the behavior from lying on its back. How soon macra turns from the state on its back to the state on its stomach is the indication for the achievement of rolling-over motion in this experiment. Fitness for one evaluation is calculated by the average of the trials in the evaluation.
Figure 11 shows fitness transition during the behavior acquisition phase. The horizontal axis is the generation in GA, and the vertical axis is the fitness. The red solid line is the best fitness in each generation, and the blue dashed line is the average fitness in each generation. Evolution stops after 189 generations, and the best gene in the 189th generation is called the acquired gene. Rolling-over motion is tested using actual macra in the real environment by decoding this acquired gene. Acquired rolling-over motion is shown in Figure 12. Figure 12(a) shows the motion in the simulation environment, and the right shows the real environment. In this experiment, rolling-over motion takes 10 s.
(a)
(b)
Soft sensor flesh used in Sections 3 and 4 is our first prototype. In the next two sections, improved soft sensor flesh is proposed. Firstly, soft sensor flesh embedded with both 3-axis force/torque sensors and integral molded sensors are presented in Section 5.
5. Soft Sensor Flesh with Integral Molded Sensors
Since soft thick exterior has enough room to embed various sensor elements in the middle of the exterior, sensors can be arranged at intended depth in the flesh as described in the basic idea of soft sensor flesh. For embedding sensor elements, we developed integral molding method for various types of sensors [17]. Required position accuracy for the sensor layout is about the same as the size of sensors and its cost is required to be low since implementation to the whole-body size flesh is considered. As one method that meets these requirements, we propose the method which is shown in Figure 13. At first, embedded sensors are glued at the tip of magnet type attachment (Figure 13(a)). After these attachments are set inside the mold by magnetic attraction, urethane foam is molded (Figure 13(b)). Sensor attachments can be removed easily after completion of the integral molding while sensors are left inside the foam. By adjusting the height of the attachment, intended depth inside flesh can be achieved. Thus integral molded sensors in soft flesh are constructed. In Figure 14, actual molding process for creating the sensor embedded torso flesh for a humanoid “macra” is shown.
(a)
(b)
Although other methods can be considered for realizing integral molding, each methods have difficulties compared with the proposed method.(i)If sensors are inserted by cutting foam after the foam are made, precise arrangement of the slits is relatively difficult.(ii)If sensors are molded after creating the sensor arrangement surface and placing the sensors on it by adding sensor locating mold, glued surfaces become hard. Also, increased number of necessary molds makes the cost higher.(iii)In the industrially used methods, there are the ones that metal parts and resin parts are molded integrally. However, there is no appropriate method for embedding sensors completely inside the foam.
By the proposed methods, we have developed second prototype of soft sensor flesh for a humanoid “macra” (Figure 15). As complementary sensors used with distributed 3-axis force/torque sensors, shock sensors (PKS1-4A-1, manufactured by Murata Manufacturing Co., Ltd.) are embedded in the flesh. Also, thick flesh can contain many other functional elements other than sensors. As is shown in Figure 15, following elements are embedded: RC servo motors for driving flesh locally, LEDs for displaying some internal states, and solenoids that expresses heart beat which can be felt during interactions in a very close distance.
For confirming sensing ability of the implemented humanoid sensor flesh system, sensor outputs were recorded while a human is holding macra up and then hugged it softly and patted it (Figure 16). In the right column of Figure 16, sensor outputs are drawn on the robot’ s geometrical model: outputs for 3-axis force/torque sensors are drawn as red thick arrows, and the ones for shock sensors are drawn as white circles. Here, centers of each white circle are located at the place where each shock sensors are embedded, and the radius for the circles is proportional to the strength calculated from the shock and frequency of the patting action. In the middle row of Figure 16, the person grabbed the robot at the side of its torso and held the robot up. There is a big red arrow in the right column of Figure 16. In this phase, outputs for 3-axis force/torque sensors at the side of its torso became high. Then the person patted the robot. The patting frequency was about 5-6 Hz. In the bottom row of Figure 16, white circles which correspond to shock sensors are big. Outputs for 3-axis force/torque sensors are small since the human hugged “macra” lightly.
(a)
(b)
(c)
(d)
(e)
(f)
In this section, functional elements are embedded in a soft flesh, and sensing and expressing abilities of a robot are improved. However, if the elements inside a flesh is rigid, softness for the exterior can be ruined. In the next section, we propose new soft multiaxis deformation sensor for solving this problem.
6. Improved Soft Sensor Flesh with Soft Multiaxis Deformation Sensors
Even if the robot’s exterior is softly made and exterior itself is suit to close contacts with humans, tactile impression can be ruined by the embedded rigid and hard tactile sensing devices. So far, there are many studies about developing soft tactile sensors [10–12]. However, these sensors are still in the developmental stage, and it is not easy to introduce those elements into our robot system. Therefore, we have developed the soft 3D deformable sensing device [18] for giving multiaxis distributed tactile sensation to our robot.
The developed sensor is shown in Figure 17(a). The basic principle of deformation sensing for this sensor is illustrated in Figure 17(b). Sensing part for infrared rays (“Light-receiving box”) is molded in the soft urethane cube, and the 3D deformation of the urethane body can be speculated from the changing value of the received infrared ray at the sensing part (influence from outer environment for infrared rays can be reduced by embedding this sensor into thick foam, a ‘‘flesh’’).
(a)
(b)
In Figure 17, arrows from “Light-receiving box” indicate normal directions for each light-receiving elements. By adopting the method to convert the deformation to the received light strength, the problem of the drift of the sensor output caused by heat, which is a big problem for capacitance-type force sensors that we used in our first prototypes, is solved. The arrangement of the light-receiving elements is very important for distinguishing multiaxis deformation. In some arrangements, received light patterns for different simulations can become almost the same. In our sensor prototype, we assume that these sensor elements are covered by soft thick exterior. Under this assumption, local deformation is not large enough to have almost same value between two different stimulus when the light-receiving elements are attached to the plane of some polygon which is referred to “Light-receiving box” in Figure 17 (it is an inclined cubic shape in this case).
In Figure 18, schematics of the internal structure of the developed sensor is shown. Developed sensing element is molded to mm cubic shape. Here, cubic shape is adopted for preventing displacement between the urethane exterior and the attached sensor elements. The size of the sensor can be smaller by selecting the mechanically small lightreceiving device (in this implementation, photo-transistor). As shown in Figure 18, 3 infrared LEDs are placed at the bottom of the sensor for ensuring the enough amount of the received light. For speculating 3D deformation of the sensor unit, light-receiving devices should be arranged as they are mounted on the surfaces facing each other. It is easy to make such structure when those devices is arranged at the top of the sensing element. The circuit board including the light-receiving part is made by FPC. After the light-receiving part is glued to the small cubic-shaped plastic box and the circuit board is fixed to the mold, the urethane sensor cube is molded by integral molding method.
The example of this sensor output is shown in Figure 19. Here, this output is corresponding to the one when the plate put on the sensor is rotated from 0 to 40 deg by 5 deg interval toward direction “A” in the top view of Figure 19. The horizontal axis of the graph expresses the rotated degree in deg, and the vertical axis expresses the output AD value (ADC has 10-bit value and FS is 1024). In the graph, 5 lines corresponds to the sensor outputs for each light-receiving devices (4 devices on the lateral face and 1 device on the bottom face). From the graph, output of the inclined side, that is the receiving device “A”, output the discriminable level compared with other 3 receiving devices that are located on the lateral face of the light-receiving cube. Although the sensor output from bottom device is relatively high against the output of “A”, it is not the problem since the bottom output is usually used independently from other 4 devices for deciding the vertical displacement of the light-receiving cube. We confirm similar results when rotating toward other directions. However, they are not shown here because of the space limitation. In the graph, it is confirmed that there is a little hysteresis for this sensor. It is mainly due to the physical characteristics of urethane foam. Although there are some materials which have less hysteresis for molding sensor elements, soft and light property of urethane foam is considered to be more important factor than other property in this paper for achieving whole-body enclosing type soft thick sensor exterior for robots.
Sensitivity and force ranges of this sensor unit can be adjusted to some extent by changing the property of the urethane foam. In our developed prototype, it can detect the applied force from 0 to about 1.4 kgf linearly, and around 1.5 kgf the sensor output reached almost the maximum value.
In Figure 20, a prototype of the sensor flesh is shown. Developed sensor cubes are embedded in the 30 mm thick urethane foam. Here, 9 sensor elements are arranged in about 10 cm × 10 cm area. In Figure 20(b), sensor output is drawn on the geometrical model. 3D deformation is expressed as the incline of the bar arranged at each sensor cube model, and the magnitude for each light-receiving device is expressed as the radius of the drawn circle. As you can see from Figure 20, pinching the sensor flesh prototype makes the 3D deformation of the sensors. Since this sensor is assumed to be embedded in the thick foam and used in the sensor distribution density like this prototype, influence of the leak infrared rays from the adjacent sensor elements can be almost ignored.
(a)
(b)
As it stands now, accuracy of this sensor is not so high, because assembling and molding process is totally handcrafted. But the important thing is that discrimination of the natural human-like contact states, not only pushing, but also contacts with 3D deformation, such as pinching or twisting, can be possible by this kind of simple structure. Also, easiness to realize the distributed tactile sensors for the actual robot is another key issue. Therefore, each 3D deformable sensor has a microprocessor for A/D conversion, and it can send a signal through SMBus Protocol.
Using this soft 3D deformable sensor elements, we have developed “macket” (Figure 21), a second prototype of our humanoid robot with soft sensor flesh [19]. Figure 21(a) shows the appearance of macket with sensor flesh and bone structure. In Figure 21(b), arrangement of the degrees of freedom is shown. Also, specification of macket is shown in the Table 1, and the part of locomotion performance of the macket is shown in Figure 22.
(a)
(b)
(c)
(a)
(b)
In Figure 23, an arrangement for the sensor elements and the actual situations of the human touch are shown. In the current setup, macket has 48 soft 3D deformable sensor elements inside of the thick urethane foam exterior. In the left of the figure, small blue boxes indicate each sensor element. Those elements are mainly arranged at both arms, torso, and head. Sensors for legs are planed to be added in the future. In the right of Figure 23, lower images are corresponded to the upper photos. For each situation, 3D deformation is calculated by comparing facing pairs of light-receiving devices in each sensor element, and the result is drawn on the robot’s geometrical model as an arrow. For the robot with distributed tactile sensors, information processing which eliminates undesirable tactile signals generated by self interferences is very important. Those self-interferences can be eliminated by constructing sensor signal output table relating to the robot’s joint angles. This method is originally proposed by Hoshino et al. for their sensor suit [3], and this method can be extended for multiaxis force distribution.
Recently, soft 3D deformable sensor elements developed by us are commercially available as ShokacCube CL manufactured by Touchence Inc., based on the license agreement between University of Tokyo and Touchence Inc. Using them, we have applied our soft sensor flesh to life-sized humanoid robot HRP-2 [20, 21]. Figure 24 shows an appearance of HRP2 with soft sensor flesh. Inside the soft sensor flesh of HRP2, there are 347 elements whole around the body. In Figure 25(b), 3D deformation vectors around front chest and right forearm regions are drawn as arrows on the geometrical model of the robot.
(a)
(b)
Our research on soft sensor flesh begins from developing prototype soft sensor flesh for achieving close safe interaction behavior between robot and human. However, such soft exterior can work to protect the robot itself. From such point of view, new structure of soft sensor flesh for self-protective behavior is proposed in the next section.
7. Soft Sensor Flesh for Self-Protective Behaviors
Soft thick exterior is not only for close interaction with humans and objects. In some case, it works for saving themselves from physical and direct impact. From this viewpoint, we develop new design of soft sensor flesh: multilayered soft thick exterior [22]. Marshmal is a robot with this new type of soft sensor flesh (Figure 26).
(a)
(b)
New soft thick flesh contains particularly flexible parts (on elbows and knees), high elasticity parts (hands and toes, which contact with the environment firstly in an accident), and shock-absorbing gel parts (inner flesh, and around precision machinery). Finally, they are wrapped by very thin film and stretchable cloth cover in whole-body surface to protect all flesh structure against wearing. It is hard to mold small flesh part with accuracy, therefore softness quality of some parts like around eyes and neckline which are not necessary for self-protection for some accidents is unwarranted. Unlike macra and macket, mold die for Marshmal is made of nylon-resinous molding dies, which is manufactured by rapid prototyping method. This new mold die has only 3 mm thickness and is very light. Easy handling and opening arbitrary injection ports over the surface enable multilayered molding.
In Figure 27(a), trial pieces for Marshmal’s right leg are shown. These pieces are foamed by urethane parts as main material, and include urethane sponge parts and shock-absorbing gel parts (shown in the top row of Figure 27(a)). Additionally, they are wrapped with soft and stretchable cloth layer, which helps very well in self-protective flesh surface against being deteriorated or worn down. In Figure 27(b), -axis acceleration in fall down experiment using this test piece for right leg with an uniform urethane and multilayered materials is shown. These trial pieces fell from height of 300 mm with 2.5 kg weight to the popular carpet. The greatest value of fall down impact decreases remarkably, and impact time simultaneously increases by using multilayered flesh cover with the shock-absorbing gel material. With this experiment, we have confirmed a usability of multilayered flesh exterior with the special gel from the point of view of self-protection against fall down impact.
(a)
(b)
At the same time, we have developed a protection mechanism set directly in robot’s multijointed structure, since soft cover can only absorb the shock to the body but not extinguish the load itself. Shock resistive mechanism, such as a torque limiter structure on rotational joints, must be introduced to protect the robot against failure by the load. So far, we have developed small torque limiter using a ratchet method for preventing from damage during interaction with humans [19, 23]. We improved our previous torque limiter for absorbing falling down shock. The size of our new torque limiter is 37 mm diameter and up to 20 mm height. In Figure 28, the structure of the developed torque limiter is shown. The two ratchet balls are each held down by the leaf springs, and easy adjustment of yield torque is available by modifying the number of leaf springs. Only when torque acts to the joint, small rolls come off, so that parent link and child link slip relatively: this is a joint dislocation. In addition to that, a potentiometer board is built into every dislocatable structure, so that measurement of dislocation angle and auto recovery are available.
In Figures 29 and 30, falling down experiment from the ordinary desk is shown. In this experiment, the height of the desk is 750 mm. When the robot detects falling down event from the change of sitting attitude angle of the pelvis, it shifts to the contact posture in preparation of falling down contact and makes all servos off. The contact pose is not to place both arms but to land by the abdominal flesh. After landing on the ground, the robot checks which joints dislocated in whole body, and recovers all dislocated joints. In Figure 30(b) shows the change of all dislocatable angles throughout falling down accident. This robot has eight dislocatable joints, and in this experiment seven joints are exposed to overload stress. As shown in the graph, dislocation recovery is executed in all seven dislocated joints. Figure 30(a) shows the impulsive force in this experiment. Robot has suffered a very large impact, even though shock-absorbing flesh works very well, and motor joints are well protected. This may indicate the limit of fixed and passive flesh structures. For improving shock absorbing function of the soft sensor flesh, some types of active control of the flesh are necessary.
(a)
(b)
In this section, combined system of soft sensor flesh and joint dislocation mechanism for absorbing falling down shock is argued. It is one challenge for adding new features to soft sensor flesh from hardware aspect. In the next section, other challenges from sensing aspects are explained. New additional soft tactile sensors which improve contact sensing ability of soft sensor flesh are proposed.
8. Additional Soft Tactile Sensors Which Improves Contact Sensing Ability
For improving tactile sensing ability of soft sensor flesh, we also develop some other types of additional sensor elements. For keeping softness of the exterior, those sensors should also be soft. In this section, two different type soft sensor elements are explained: one is a stretchable knit sensor [24], and the other is a 4-axis soft deformable sensor [25].
8.1. Soft Sensor Flesh with Stretchable Knit Sensor
One of the most difficult part of the body for acquiring tactile information is joint region. Since the degree of the change for joint region is large, soft and stretchable tactile sensor element is necessary for covering joint region. Proposed soft sensor flesh itself also has this problem. For solving this problem, additional stretchable sensing element is developed for detecting tactile sensor change around joint region.
So far, there are some studies for constructing soft and stretchable sensing structure. For example, Hoshi and Shinoda have developed a touch-area-sensitive tactile sensing element [11], where stretchable conductive fabrics are sandwiched by two compressive insulators having different softness. Although their sensing element has softness and stretchable properties, those parameters largely depend on the mechanical properties of the compressive insulators, and they do not discuss the feasibility for their sensor element to implement it as a humanoid’s exterior. Also, Tada et al. have developed flexible and stretchable tactile sensor element based on static electricity phenomenon [26]. Although their sensing element is made of conductive silicone rubber and it can be stretched until 300% of the original length, their sensor can detect only touching and releasing events, not the pressure. Nagakubo et al. have developed wire-saving stretchable tactile sensing element [6] using EIT (electrical impedance tomography) technique based on inverse problem theory. In their method, pressure distribution can be calculated by measuring potential differences of the electrodes which is arranged around the boundary of the measuring region. They have tested 150% stretchable conductive knitted fabrics which are made by spraying water-based carbonic paint over cotton knit. However, such complicated structure can limit the stretching property, and the cost for making whole-body enclosing type exterior is not so easy.
Therefore, we adopt Hoshino et al.’s method for constructing our stretchable sensor element, and extend their method for making it more stretchable and getting the analog continuous pressure value at each sensing points by using pressure sensitive knitted electro-conductive yarn. Figure 31 shows the sensor structure (a) and a test piece of the sensor element (b).
(a)
(b)
As shown in Figure 31, there are six layers in the developed sensor element and each layer is manufactured as follows.(1)Outer and inner insulating layers: these layers are made for protecting the internal layers and knitted by insulating yarn using single-rib knitting method.(2)Wiring layer: soft thin coated copper wires are embroidered on a insulating structure knitted by single-rib knitting method. For keeping the stretchability of the knit structure, wires are sewed with a half-back stitch.(3)Electrode layer: in this layer electrodes connected with signal wires from wiring layer are attached on the insulating knit. For electrodes, small brass circular plates (diameter 8 mm, thickness 0.4 mm) are used, and the terminals of the signal wires are soldered to the electrodes.(4)Resistance layer: high resistant conductive strings are knitted by single-rib knitting method. For making a pressure sensitive structure whose resistance uniformly changes in proportion to the contact force, this layer should be knitted only with the high resistant conductive strings. Although the strings used here requires trained skill to knit them, easier method like double knitting of these strings and normal insulator strings deteriorates the pressure sensitiveness remarkably.(5)Ground layer: this layer is connected with the signal ground.
Next, relationship between the applied force and output voltage is examined. In Figure 32, output voltages for developed knit sensor are shown. In this experiment, applied force is increased at about 4 kgf/sec via flat circular metal parts having 16 mm diameter. In Figure 32 horizontal axis indicates the applied force in kgf while vertical axis expresses the output voltage in V. There are 3 different stretching conditions plotted in the graphs. Those are no stretching condition and stretching conditions toward two directions. For knitted fabrics, those two directions are defined as wrap direction and filing direction. In the single-rib knitting method, stretching degree toward filling direction is much larger than the one toward wrap direction. In Figure 32, the red ‘‘+’’ signs express no stretching condition while green ‘‘’’ signs and blue signs ‘‘’’ indicate stretching condition toward filling direction and the one toward wrap direction, respectively. The maximum stretching length is 200% toward filling direction while the one toward wrap direction is 150%.
Although the output voltage changes nonlinearly as an applied force increases, it is confirmed that the knit sensor can output continuous analog value in all measured conditions from the graph. Also, the load-voltage properties are affected by how long and which direction the sensor element is stretched. When it is stretched toward filling direction, output voltages are smaller than the ones for nonstretched condition. This can be due to the different density changes of the resistance layer and ground layer. Since the size of the string for ground layer is smaller than that for the resistance layer, ground layer becomes more sparse than resistance layer. As a result, resistance around contact region becomes larger compared with the no-stretched condition, and output voltages become smaller in such situation as described above. On the other hand, sensor output voltages become a little bit larger than no-stretched condition when it is stretched toward wrap direction. This can be also due to the different density changes for the resistance layer and ground layer. Ground layer becomes dense when it is stretched toward wrap direction. As a result, resistance around contact region becomes smaller compared with the no-stretched condition, and output voltages become larger. Also, it is confirmed that the sensor output becomes easier to get maximized when the applied force is larger than about 1 kgf in the filling stretched conditions. This is partly because the fabrics both in resistance layer and ground layer tangle each other when they are pressed strongly.
Using developed analog knit sensor, sensor outputs while human touches the sensor in a various way is examined here (Figure 33). In this experiment, knit sensor is not stretched and fixed on a table. As shown in Figure 33, temporal sensor output patterns differ from each other, and at least the contact state examined here can be discriminated from the shapes of the sensor outputs. For example, the difference between “push softly” and “push strongly” can be recognized clearly. Also, developed sensor can respond against relatively weak contacts like “stroke” and “tickle.” Furthermore, it can sense impulsive pattern like “pat.” Using proposed method, cardigan type knit sensor exterior that covers soft sensor flesh is developed (Figure 34). The developed cardigan knit sensor is created for macket. In Figure 34, (a) shows the developed cardigan knit sensor while (b) shows how the sensor is implemented on the humanoid. Here, processing boards for this sensor are put in a backpack. For this knit sensor, the inner insulator layer is omitted since macket has already soft urethane exterior. Here in this case, the cardigan knit sensor is expected to work as add-on type functional exterior. Without this knit sensor, macket cannot detect weak contacts with its own tactile sensor flesh. The developed cardigan knit sensor has 75 electrodes. Electrodes at the front body and back body are arranged uniformly as a whole. For both arms, electrodes are arranged mainly around shoulders, elbows, and wrists since these parts are expected to be touched by humans frequently during the interaction. In the current situation, sensor calibration is manually done by checking load-voltage properties for each sensor electrode.
(a)
(b)
(a)
(b)
In Figure 35, some of the example reactive behaviors based on the sensation of the knit sensor are shown. In the graphs of each figure, only data from electrodes which output values beyond a threshold voltage are drawn. In Figure 35(a), electrodes for left elbow (Electrodes 33 and 34) detect “soft push” while an electrode for right elbow (Electrode 69) detects “strong push.” But, reaction behavior against the “soft push” for left arm (watching left arm) is generated since the soft push is detected earlier than the strong push.
(a)
(b)
On the other hand, “stroke,” defined as weaker outputs than “soft push”, is detected for left arm’s electrodes (Electrodes 24, 25, and 26) in Figure 35(b). After detecting the stroke, nodding, a reactive action against “stroke” in this situation, is generated as shown in the photo.
8.2. 4-Axis Soft Deformable Sensor for Detecting Twisting Deformation
With the soft 3D deformable sensor element we have developed, twisting deformation cannot be detected by one sensor unit. In some parts of the body, detection of the twisting deformation of the sensor exterior can be used as some social signal from a person that the robot communicates with.
Therefore we propose a new design of soft deformable sensor. Figure 36(a) shows the basic structure of the newly designed sensor, ‘‘4-axis soft deformable sensor.’’ This sensor consists of two basic elements: a pair of IMUs (Inertia Measurement Unit) and 1 force sensor. On both of a top and bottom IMU board, one 3-axis MEMS accelerometer and one 3-axis gyro-sensor are implemented. By sandwiching a soft urethane foam with two IMU boards, soft 3-axis rotational deformation of this sensor unit can be detected from the output differences between two IMUs. Also, 1-axis compressive deformation is detected from the 1-axis MEMS force sensor arranged at the bottom. In total, this sensor can detect 4-axis soft deformation. Especially, by detecting the 3-axis rotational deformation from the differential output between two IMUs, sensor changes due to the externally applied force to the internal frame, or the subtle vibration of the actuator can be canceled. This leads to a robust sensing of the soft deformation of the sensor unit. By the proposed sensor design, detection of twisting sensation by one sensor unit is possible by estimating rotational deformation from 3-axis acceleration and 3-axis angular velocity.
(a)
(b)
For online estimation of 3D orientation from both top and bottom IMU boards, a method using extended kalman filter [27] or a method using complementary filter [28] can be applied. Here, quaternion for 3D orientation of the sensor unit is estimated using extended kalman filter. In our sensor design, drifting problem of the estimated result can be solved. Since two IMU boards are connected with elastic urethane foam and the relative position is getting back to the original position without any external force, estimated quaternion is reset to the original posture when no external force is monitored.
In Figure 36(b), developed prototype of 4-axis soft deformable sensor is shown. The external dimensions of this sensor prototype is 16 mm × 16 mm × 25 mm. Its weight is 6.7 g. For this prototype, we developed common IMU boards for estimating 3D rotation of top and bottom plane of the sensor unit. Although IMU boards developed here are rigid PCB, softness of the sensor unit is not ruined since sandwiched urethane foam has enough thickness. For detecting normal direction force, we developed another circuit board and arrange it under the bottom IMU board. Motion of the bottom IMU board and attached force sensor board is constrained only to the normal direction by plastic board container. For urethane foam, we use commercially available semirigid polyurethane foam called EMT (density is cm3, elasticity is less than 30%. Hardness is less than 14.71 kPa), manufactured by Inoac corporation.
Evaluation experiment for investigating twisting sensation ability of developed 4-axis soft deformation sensor is done. Figure 37(a) shows experimental setups. In the experiment, we use magnetic 3D motion tracking sensor, Fastrak manufactured by Polhemus, for evaluating our sensor prototype. Since Fastrak has 0.8 mm of position accuracy and 0.15 deg rotational accuracy, it can be used as a true value of 3D rotation for our sensor prototype. Upper IMU board is connected with a receiver of Fastrak while bottom surface of the sensor prototype is fixed to a desk. Rotation against -axis of the coordinate system in Figure 37 is measured at 30 Hz. During the measurement, the sensor prototype is kept still for about 3 seconds. This is because average values of initial 100 samples are used as offsets of gyro sensors. This process is an only necessary calibration for 3-axis rotational deformation. This calibration process is largely simplified compared with the previous 3-axis soft deformation sensor.
(a)
(b)
Measurement result for -axis rotation is shown in Figure 37(b). In Figure 37, horizontal axis indicates time in seconds while vertical axis expresses rotation in degree. Red solid line expresses the output from Fastrak while green dashed line expresses the estimated output of the developed sensor prototype. Although overall behavior of the developed sensor prototype is similar to the one of Fastrak, the output of the sensor prototype has 200 ms delay and about 10 to 15 degree estimation error in maximum. When rotational changes from 0 to peak value is compared, the output of the sensor prototype is about ±24%.
For testing feasibility of the developed sensor, simple soft sensor flesh which contains 4-axis soft deformation sensor is developed for a small humanoid robot NAO, manufactured by Aldebaran. In Figure 38, developed sensor flesh and its geometrical model are shown. In Figure 38(a), inside of the exterior is shown. This prototype exterior consists of two urethane parts and two 4-axis deformation sensors. Two deformation sensors are arranged at each side of the torso. In Figure 38(b), NAO with developed soft sensor flesh is shown. Also, geometrical model for NAO with this sensor flesh is shown. This model is used for behavior generation and online sensor view in the experiment. At the bottom left of Figure 38(b), magnified image of the sensor model is shown. This sensor model shows the 3D rotational deformation by the orientation of the three perpendicular arrows, and magnitude of normal direction force is shown by the length of the pink arrows and color of the sensor box.
(a)
(b)
Reactive behavior against tactile sensation of the side of the trunk is implemented for an experiment. NAO starts wriggling only when both outputs of the sensor show the deformation toward front inside of the body: that is, if the sensor outputs for an -axis of the sensor local coordinates go beyond a positive threshold by human inputs, predefined motion sequence is replayed by the robot. Result of this experiment is shown in Figure 39. As you can see from the photos, it succeeded in reacting human’s contacts on its side of the torso. Detailed outputs of the embedded deformation sensor on the right side of the torso are shown in upper part of Figure 39. In the graph, horizontal axis indicates time in seconds while vertical axis expresses rotation in degree. Three lines corresponded to each axis of the sensor local coordinates are drawn at the left of the graph. In the graph, rotations for both -axis and -axis show positive values. Here, positive rotation around -axis expresses forward deformation of the flesh while positive rotation around -axis indicates deformation toward inside of the body. This means that human touch was correctly measured in this experiment.
9. Conclusion
In order to achieve close interaction behavior of robots against humans, objects, and environments, the idea of “Soft Sensor Flesh” is proposed. Unlike other ordinary artificial skin with distributed tactile sensor network, soft sensor flesh consists of soft and thick foam and many tactile sensor elements embedded inside the foam including multiaxis deformation detectable sensors. In this paper, several types of prototype of soft sensor flesh are shown, and multiple humanoid robots with those soft sensor flesh are developed. Furthermore, feasibility of the soft sensor flesh is confirmed through both actual behavior experiments and computer simulation.
One of the important challenges for the next step is the information processing of the enormous amount of tactile sensor information, which is now being worked on by the authors [29]. Also, using developed robots with soft sensor flesh for concrete targets, such as nursing care task, is another important challenge. Knowledge accumulated by using soft sensor flesh for various practical tasks will accelerate the study for improving soft sensor flesh itself.