Abstract
The interactive nanomanipulation platform is established based on fuzzy control and connected region marking (CRM) algorithm in SEM. The 3D virtual nanomanipulation model is developed to make up the insufficiency of the 2D SEM image information, which provides the operator with depth and real-time visual feedback information to guide the manipulation. The haptic device Omega3 is used as the master to control the 3D motion of the nanopositioner in master-slave mode and offer the force sensing to the operator controlled with fuzzy control algorithm. Aiming at sensing of force feedback during the nanomanipulation, the collision detection method of the virtual nanomanipulation model and the force rending model are studied to realize the force feedback of nanomanipulation. The CRM algorithm is introduced to process the SEM image which provides effective position data of the objects for updating the virtual environment (VE), and relevant issues such as calibration and update rate of VE are also discussed. Finally, the performance of the platform is validated by the ZnO nanowire manipulation experiments.
1. Introduction
As an assistant imaging tool to research nanomaterial characteristics and the visual detection device for nanomanipulation, SEM has been paid more and more attention for its real time imaging and large operation space [1, 2]. Compared with AFM, SEM is used as a tool for providing the operator with real-time visual feedback during the nanoscale manipulation although it only provides the 2D images of single view angle [3], which contains limited environment information. For example, Sitti [4] has established the SEM-based robotics systems for Microscale and nanoscale with AFM as the handle tool; Fatikow et al. [3, 5] has realized the automatic nanohandling inside SEM using the advantage of vision feedback; Penga et al. [6] did a lot of experiments to study the probe in the SEM. In the existing SEM-based nanomanipulation system, the operator could not judge the position relationship among probe, objects, and substrate accurately only according SEM image, which would easily damage the devices and objects [7, 8] as does in open loop during manipulation. Due to the unstructured character of nanoenvironment and scale effect of nanomanipulation, micro force sensor is a possible solution compared with position sensor. On the other hand, force sensors for micro/nanomanipulation, especially for nanoscale, are difficult to manufacture and expensive, which is only suitable for force/haptic information in contact status [9]. Therefore, it is particularly important to introduce the virtual reality technology for the unstructured nanoenvironment [10]. It can guide and control the real nanomanipulation platform working in the master-slave mode with virtual nanomanipulation model and virtual force feedback information. It is a novel approach to realize the nanomanipulation with real time, accuracy, high efficiency and friendly human-machine interaction.
This work is motivated for manipulating the nanocomponent in SEM with telepresence as in macro scale. By adopting the virtual reality and haptic technology, the operator can handle the nanowire in SEM by controlling the virtual force. At the same time, although the SEM can only offer 2D image, the operator can feel the force and 3D visual information offered by virtual reality technology. The main motivation of this work is to establish a SEM-based master-slave telenanomanipulation platform having the performance of security, reliable, and real-time without force sensor.
With the aid of force feedback device (Omega 3), the nanocomponent manipulation method combined with virtual reality and SEM is studied. It can provide users with senses of force and more intuitional operation interface, as well as changeable view point and angle in virtual environment (VE) to provide more sufficient visual information of nanomanipulation in previewing and real time tracking. While updating the VE, the CRM algorithm is introduced to process the SEM image.
The slave (Attocube) needs relatively stable control variable to avoid damage of the probe, nanowire, and substrate, which is controlled by the master (controlled by the operator). But the operator’s input is so unstable that it is difficult to control and accurately formalize mathematically. Fuzzy logic has become a particularly widely used methodological approach on real world applications control. Fuzzy logic-based systems do not require models, which make them especially appropriate for processes whose mathematical formalism is not clear or global for all the cases [11, 12]. The systems with fuzzy characters are studied extensively [13–16]. The fuzzy control theory and its improved theoretics have been used in many practical applications [17], such as in noisy image segmentation [18], languages character recognition [19], and prioritizing service attributes [20]. Based on this, in the literatures, the fuzzy control is adopted in the master-slave control.
The organization of the paper is as follows. Section 2 describes the designed system framework in detail, including control information and data exchanging procedure, the fuzzy control for the system. The creation method of grid model in VE as well as dynamic modeling of probe and nanowire using skeleton sphere is discussed in Section 3. Section 4 demonstrates the collision detection method using Hierarchical Bounding Volumes (BVH). Force rendering model is built to embed force information to the VE in Section 5. The region marking algorithm (CRM) used to process SEM image is introduced in Section 6, and the procedure of VE update is also illustrated in this section. In Section 7, the performance of the platform is validated on experimental results via nanowire manipulation experiments, and the results are also analysed. Section 8 is the conclusions and the novelties of our platform.
2. System Construction
The overall structure of the master-slave nanomanipulation platform is displayed in Figure 1. The platform is mainly composed of the following components: tungsten probe, nanopositioner (Attocube), SEM vacuum manipulation environment, master haptic device (Omega 3), and the master/slave control PC. The tungsten probe is installed on the nanopositioner, working in SEM vacuum environment as the slave. The slave PC controls the imaging of SEM and communicates with the master control PC for real time image transmission via the TCP/IP. The master control PC interacts with the operator, running a 3D virtual nanomanipulation environment and the main control interface with transmitted SEM image integrated. The 3-DOF haptic device communicates with the master PC through USB. The three axes of the Attocube nanopositioner are controlled with its dedicated controller (ANC150) in remote way by the master control PC according to the information acquired from Omega 3. The nanomanipulation platform presented in this paper could accomplish multi-DOF nanomanipulation for various kinds of nanocomponents with the help of SEM real-time visual feedback.
As the master, the haptic device Omega 3 has energy output to outside. If the haptic interface composed of Omega 3 is unstable, damage will be caused to the manipulation object and the manipulation tool. Meanwhile, the transparency of manipulation process will be destroyed. For the adjustment process from the control of the operator to the suitable control variable needs time and the model of the operator is hard to establish, the robustness of the fuzzy control is considered. Therefore, fuzzy control unit is added to control the haptic interaction system, which is composed of operator, haptic interface, and the virtual environment. Energy conversion between the operator and the virtual environment is completed by the haptic interface.
The input of the fuzzy controller is the deviation and deviation change rate of the output force of the operator (that is the displacement of the Omega 3). The output is the controlled variables transmitted to virtual environment, which are transmitted to Attocube Piezo Step Controller after large scaled down. In the manipulation process, the detection of and is done continuously and the parameters modification online are conducted according to the fuzzy control principle, in order to meet the requirements of the control parameters and to improve the output characteristics of virtual manipulation.
According to the characteristics of Omega 3 and the haptic control manipulation, the action range of the fuzzy controller is selected objectively. The range of the error and error change rate is defined to be the universe of the fuzzy sets:
Its fuzzy subset is
Assuming that they all obey normal distribution, the quantization factor of and is 1.
The weighted average method with smoothing output inference rules is adopted. The crisp value of the weighted average of the output of the membership degree is picked up. The final output value of the fuzzy inference is determined. That is, the areas bounded by the fuzzy membership function cure and the horizontal ordinate are the final output value.
3. Establishment of Virtual Nanomanipulation Model
Virtual nanomanipulation is an effective way to provide the 3D visual information and simulation for real nanomanipulation, which is helpful for the operator to judge in real time, enhance their perception, and proceed off-line simulation [21–23]. The credibility of simulation interface depends on the virtual operating model. In this sense, the more realistic of the operated object model, the more actual operation scene will be reflected in the simulation interface of virtual vision. Meanwhile, the precision of operating model reduces corresponding with the increasing of modelling error in the nanoVE.
As a software interface of graphics hardware, OpenGL can satisfy modelling demands with advantages of fast getting state, good characteristics of base development, and dynamic display. Besides, it is convenient to apply in other engines with its independent hardware interface and variety library functions. Therefore, OpenGL is adopted as the base development tool to establish the basic model of nanowire in this paper. Aiming at establishing the virtual nanowire model, vertices are drawn as grid elements and formed mesh model using OpenGL functions. Although the mesh model can be edited to realize geometrical deformation, it is important to know which grid element needs to be deformed. Because of the large calculated amount of analysing each grid element, a kind of simple assistant model is needed, whose characteristics are analysed to determine the rules of deformation.
In this paper, skeleton model is adopted as the simple assistant model to research the deformation of the virtual nanowire model. Skeleton models are widely used in creating dynamic model as the role of bone in animals and plants, and the skeleton in 3D model is used to judge the motion property. A certain amount of skeleton models would be filled in the mesh model (virtual nanowire model) built with OpenGL. According to the dynamic characteristic of the nanowire, skeleton spheres are suitable to fill the nanowire model. The skeleton sphere is characteristic sphere. By judging the position of every skeleton sphere in VE, the vertices is reedited to realize the dynamic deformation of the virtual objects, namely, that the mesh model (virtual nanowire model) would change its shape as the skeleton spheres. Simple perspective map of nanowire model filled with skeleton spheres is shown in Figure 2. Sketch map of modeled nanowire filled with skeleton spheres in VE is shown as Figure 3.
Figure 4 is the deformation schematic diagram of skeleton spheres in VE. It is shown that the relative coordinates of polygon grid elements also change as the coordinates of skeleton spheres change. Deformation model of skeleton spheres in VE is shown in Figure 5. In order to see the movement relationship between skeleton spheres and nanowire more clearly, the updating speed of movement is reduced; thus, the effect of nanowire motion driven by skeleton spheres is more obvious.
4. Collision Detection
In order to perceive the immersive of presence and complete interactive force in the VE, fast and accurate collision detection between different models is necessary [7]. Collision detection is to detect whether different objects in the VE have bumped into each other. In fact, two impenetrable objects cannot simultaneously occupy the same space in the real world; however, they can occupy the same space in the virtual world. In order to avoid regional mixed taken by the objects cross in the virtual world and offer the sense of reality, collision detection is needed [24]. Besides, the data of virtual force presence mainly comes from the collision detection among different objects. When designing a collision detection system, the design factors, namely, all aspects of developing collision detection modules, should be firstly considered in collision detection algorithm. There are many collision detection problems worthy of further research. In this paper, the structures of objects to be manipulated are relatively simple, so we only analyze the collision detection suitable for this platform. In other words, the algorithm we studied is suitable for the collision detection between single cone probe, columnar nanowire/nanotube and the substrate with smooth surface, and good rigidity. It can be used in the force presence of nanowire/nanotube manipulation handled with single probe based on adhesive control.
Among the collision detection algorithms, we usually do not adopt the “full-featured” collision detection method, while selecting the specific collision detection method is an advisable choice [25]. For the simple nanowire model in this system, BVH method is more suitable. But it is different from the collision detection algorithm of static models, the collision detection between probe and nanowire is normally dynamic and more complex. Thus, the difficulty of collision detection is greatly increased. Hence, the collision detection of nanomanipulation is divided into three parts: between probe and substrate, nanowire and substrate, and probe and nanowire, then they are integrated.
The mesh model of the probe is built with OpenGL according to the geometric parameters obtained by SEM image. Based on the virtual probe mesh model established, suitable bounding spheres which are as one category of BVH according to pyramid (the probe model) data are filled in the probe mesh model to realize the intersect detection between probe and nanowire. The arrangement method of bounding spheres filled in probe model determines the effect of collision detection between the virtual nanowire model and virtual probe model. Normally, there are three approaches to arrange the bounding spheres in probe mesh model: sphere diameter arrangement (SDA), prestack sphere center arrangement (PESCA), and poststack sphere center arrangement (POSCA) (different arrangement methods are shown as Figure 6). According to the feature of the probe (large cone height) and experimental comparison among the three arrangement method, the PESCA method is adopted since it neither leaves out a lot of empty space in probe mesh model as SDA method dose nor makes complex permutation order and larger calculated quantity as POSCA method dose. Figure 7 shows the collision detection between probe and nanowire by using PESCA method. It can be seen from Figure 7 that the probe is filled properly, and the collision between the probe model filled with bounding spheres and the nanowire model could be detected, which prevents penetration between models.
(a) Sphere diameter arrangement (SDA)
(b) Prestack sphere center arrangement (PESCA)
(c) Poststack sphere centre arrangement (POSCA)
5. Force Rendering Model
In the whole nanomanipulation process, embedding of virtual force feedback as a kind of force sensing can greatly enhance the operator’s perception [26], which is important and essential for nanomanipulation and nanoassembly [27, 28]. It can allow the operator to carry out simulation of nanomanipulation better and guide real nanomanipulation more effectively.
The force rendering provides an interactive approach through which the operator could find out what kind of effects was produced between the virtual models. In nanoscale manipulation environment, the forces among operating objects, operation tools and the substrate are complicated. For SEM vacuum working space, the Van der Waals force acts as a dominated force among the nanocomponents; therefore, the force rendering is conducted based on Van der Waals force.
5.1. Force between Probe and Substrate
The shape of the probe tip is simplified as two parts [29, 30]: microsphere and microcylinder. According to Hamaker’s assumption, the forces between each part and the substrate are calculated, respectively, and then added up to create a resultant force.
The interaction energy between the substrate and the microsphere molecule of the probe tip could be expressed as where is Van der Waals constant, is the numerical density of the substrate, and is the minimum distance between the substrate and the microsphere. And the interaction energy between the substrate and all molecules of the microsphere could be obtained via integral calculation of (5.1): where is the numerical density of the microsphere, and is the height of the microsphere.
The force between the substrate and the microsphere is obtained via differential calculation of (5.2): where is Hamaker constant.
Similarly, the force between the substrate and the microcylinder is calculated as where is the height of the microcylinder.
Finally, the force between the substrate and the probe tip could be expressed as
5.2. Force between Nanowire and Substrate
The force between the substrate and the nanowire could be viewed as a kind of interaction force between serial particles and infinite plane, based on which the Van der Waals force between a single nanowire molecule and the infinite substrate could be expressed as the following equation according to the differential of (5.1):
The force between unit length nanowire and the substrate could be obtained by calculating integration of (5.6): where is the minimum distance between the substrate and the microsphere, is the molecule radius of the nanowire, and and are numerical densities of the nanowire and the substrate, respectively.
5.3. Force between Probe and Nanowire
The force between the probe and the nanowire could be expressed as Van der Waals force between the nanowire and microsphere of the probe tip. There exist two situations to be explained.
When the probe tip contacts with the nanowire vertically or the contacting angle is closed to 90°, the force could be expressed as the interaction between two microspheres, and the Van der Waals energy is where is Hamaker constant, is the minimum distance between the two microspheres, and and are the radius of the two microspheres, respectively. The force between the two microspheres is obtained by calculating differential of (5.8):
When the probe tip contacts with the nanowire horizontally or the contacting angle is closed to 0°, the two objects contact as parallel cylinders, and the Van der Waals energy is expressed as
And the calculated force is where represents the contact length between probe and nanowire.
The forces among the virtual models are approximately simulated according to the previous expressions when different objects (nanowire, probe, and substrate) interact with one another. And these simulated forces output by CHAI 3D engine are used to strengthen the operator’s perception when manipulating the nanocomponents.
6. SEM Image Processing and Virtual Environment Update
It is an essential step to acquire the position information of objects (probe and nanowire) in real nanomanipulation working space for updating the VE. Since no nanoscale sensors are installed in the vacuum space of SEM, the real time SEM image is considered as the only visual servo for the whole system, which is the source of the objects coordinates information. Therefore, it is necessary to research an effective SEM image processing approach to obtain the position data of the objects in the SEM image, which provides information for VE update.
Some researchers have proposed some novel algorithms about 2D positioning under SEM according to a single 2D SEM image [1, 12]. However, most of these have complex processing procedure which inevitably increases the calculation as well as difficulty of realization. In order to provide valid position data for VE updating, a connected region marking (CRM) algorithm for single transmitted SEM image is carried out in this paper to extract the figure of the objects, and then the desired information is obtained. After the SEM image is processed with gray-scale algorithm, dynamic threshold binarization and denoising algorithm in sequence, certain connected regions are marked with tags (such as number 1, 2, and 3). Then, the pixel number of every connected region is calculated. The essential regions of nanowire and probe are extracted respectively after setting the thresholds of pixel number according to the characteristic of nanowire and probe, which are used to calculate the key geometric parameter (position) in image. This method has the advantage of easy to be implemented.
The purpose of updating the VE or adjusting the virtual models is to realize the synchronization of real nanoworking space in slave and VE in master. It needs to calibrate the VE, namely, the relationship between a SEM image and VE should be determined. The detailed calibration process is as follow.
The size of the VE designed in this paper is set as , where (mm); the range of coordinates is set to , where (mm). Suppose one coordinates of the nanowire endpoint is obtained from the processed SEM image with resolution of 1024 × 768 using CRM algorithm and define , ; therefore, the coordinates of the nanowire model in VE corresponding with is expressed as , .
In addition, the rate of the VE needs to be discussed during the continuous operation. In this system, whether the VE should be updated depends on the processing results of the SEM image. The rate of SEM image transmission is determined in advance (transmission interval is set as 2 second). The system compares the current position data of objects acquired from the transmitted SEM image with last data acquired 2 second ago and stored in RAM. If the data changes beyond the given error range, the position data of the virtual models would be adjusted to the current data, which would be stored in the RAM for next data comparison. However, in the case that the data are same as last or change within the error range, the VE would not be updated, which could save the system resource and increase the efficiency of manipulation.
The SEM image processing procedure using CRM algorithm, as well as the update of VE, is illustrated in Figure 8. It can be seen from Figure 8 that the position of nanowire and probe in VE keeps synchronous with that in the original SEM image.
7. Experiments and Results
The experimental platform is shown in Figure 9, containing the SEM (KYKY-EM3200), Omega 3, Attocube nanopositioner and controller, and master and slave control PC. All the experiments mentioned in this paper have been done in the ten thousand level clean-room of Science Park in HIT.
The control interface of the experimental platform is shown in Figure 10. The 3D graphic interface in the master control PC is programmed with VS 2008 combined with OpenGL and CHAI 3D. The virtual nanomanipulation environment and the real SEM nanomanipulation image sent from the slave PC are displayed in the same interface. There are two different control modes: single-step and real time tracking. In single-step control mode, the virtual models have been controlled to move to a desired position, then the Attocube nanopositioner moves after receiving the execution commands sent by the master PC. Satisfaction test before real manipulation in single-step mode is an approach to improve the operation accuracy, although it has low efficiency. On the other hand, in real time tracking mode, the operator could manipulate Omega 3 to move Attocube nanopositioner continuously with high efficiency compared with single-step mode, but it could cause maloperation.
In the experiments, the operator pushes the bar of the 3-DOF haptic device (Omega 3) at a certain velocity to a position; the force and position information applied on the haptic device captured by the master PC is transmitted to the virtual nanoenvironment; thus, the 3D graphic interface run in the master control PC changes real time to track the movement of master; the slave Attocube positioner moves following the master as soon as receiving the position data of large scaled down. Simultaneously, the operator can feel the force in hand according to the force and position information provided by VE.
The performance of the virtual force feedback master-slave telenanomanipulation platform is tested with the ZnO nanowire (with radius of 100~200 nm) manipulation. The experimental results are shown in Figure 11. The curves of probe position during the experiment without and with fuzzy control are shown as Figure 12.
(a)
(b)
From the curves, we can find that the trajectory of the probe controlled with fuzzy control is much smoother than that without it, and it is confined to a certain scope of the starting point. That is the redundant path of the probe is avoided, and the stability and safety (avoiding the damage of probe, nanowire, and substrate, which are expensive and need much time while replaced) of the whole manipulation process is guaranteed.
8. Conclusions
In this paper, the nanomanipulation platform with virtual 3D visual and virtual force feedback is established combined with virtual reality technology and SEM. The fuzzy control algorithm is adopted to assure the system stability and safety. The virtual models of probe and nanowire are built to provide 3D visual information of SEM-based nanomanipulation. To obtain the sense of immersion and interactive force for the operator during nanomanipulation, the collision detection model of nanomanipulation and the force rendering models are built. With employing of the VR and haptic technology, the operator can handle the nanowire by controlling virtual force, so the operator’s perception to nanomanipulation is enhanced.
The CRM algorithm is proposed to process the transmitted SEM image which is the basis of VE update, and the calibration of the VE. In addition, the updating rate of VE is also discussed. The 3D visualized manipulation environment cannot only provide the operator with real time image information, but also make the platform run automatically after the satisfied virtual preview result. It could reduce the research cycles and research cost of nanomanipulation.
System performance is evaluated by the ZnO nanowire manipulation experiments. The experimental studies show that the nanowire can be accurately pushed and manipulated on the substrate using the newly developed platform. The real time visual display coupled with the real time force feedback provides a telepresence environment in which the operator cannot only feel the interaction forces but also observe the real time changes of the nanoenvironment.
Through experiments, we conclude that the platform presented in this paper has the following advantages: (1) the man-machine interactive performance is enhanced with combining of virtual reality environment, SEM feedback image, and haptic device; (2) the nanomanipulation efficiency stability and safety are improved for the simultaneous monitoring and manipulating and the employing of fuzzy control; (3) the real time performance of the nanomanipulation is improved, for the dynamic refresh of VE is realized.
In future, we will do further study on automatic detection for the contact between tip and substrate based on 2D SEM image and automatic nanomanipulation based on virtual nanomanipulation model.
Acknowledgments
This work was supported by the National Natural Science Funds for Distinguished Young Scholar (Grant no. 50725518), National Natural Science Funds for Young Scholar (Grant no. 51105117), Program for New Century Excellent Talents in University (Grant no. NCET-08-0170), Program for Changjiang Scholars and Innovative Research Team in University (Grant no. IRT0915), China Postdoctoral Science Foundation funded project (Grant no. 20100471051), State Key Laboratory of Robotics and System (Grant no. SKLRS-2010-MS19) and Harbin Science & Technology Innovation Talent Study Special Fund Project (Grant no. 2012RFQXG075).