Scanning / 2017 / Article
Special Issue

In Situ SEM Nanomanipulation and Nanomechanical/Electrical Characterization

View this Special Issue

Research Article | Open Access

Volume 2017 |Article ID 3982503 |

Chao Zhou, Lu Deng, Long Cheng, Zhiqiang Cao, Shuo Wang, Min Tan, "Automated Axis Alignment for a Nanomanipulator inside SEM and Its Error Optimization", Scanning, vol. 2017, Article ID 3982503, 8 pages, 2017.

Automated Axis Alignment for a Nanomanipulator inside SEM and Its Error Optimization

Academic Editor: Yajing Shen
Received15 Mar 2017
Accepted08 May 2017
Published19 Jun 2017


In the motion of probing nanostructures, repeating position and movement is frequently happing and tolerance for position error is stringent. The consistency between the axis of manipulators and image is very significant since the visual servo is the most important tool in the automated manipulation. This paper proposed an automated axis alignment method for a nanomanipulator inside the SEM by recognizing the position of a closed-loop controlling the end-effector, which can characterize the relationship of these two axes, and then the rotation matrix can be calculated accordingly. The error of this method and its transfer function are also calculated to compare the iteration method and average method. The method in this paper can accelerate the process of axis alignment to avoid the electron beam induced deposition effect on the end tips. Experiment demonstration shows that it can achieve a 0.1-degree precision in 90 seconds.

1. Introduction

Nanoscale manipulations inside scanning electron microscope (SEM) have found many uses in various fields [1]. Examples include nanomaterial characterizations [29], nanoelectronic probing [1012], nanodevices prototyping [13], photonics [14, 15], and biological researches [1618]. The most common setup involves installing piezo-based manipulator into an SEM. A joystick is used to control the motion of the manipulator, and the SEM provides the real-time image feedback. This combination provides an intuitive hand-eye coordinated method to interact with objectives and in micrometer and nanometer scale.

Several SEM based nanomanipulation systems have been reported in the literature [1923]. There are also commercially available systems from Kleindiek, DCG Systems (Previously Zyvex), SmarAct, Klocke, and Attocube. The majority of these systems do not have position sensor integrated, thus repeatable motions cannot be made, and the manipulation efficiency relies heavily on the skill of the human operator. Other systems contain optical encoders for position feedback, but the heat generated by the laser diode is difficult to dissipate inside vacuum, leading to high position drift rate. The use of mechanical sliding rails for guiding the piezo stick-slip motion is not repeatable due to frictions in the interfaces and the deformation in the mechanical rails. The stick-slip motion also creates mechanical vibration while in motion, which can cause end-effector (the device at the end of a robotic arm, such as claws and needles) or sample damage.

We have previously reported a new load-lock compatible nanomanipulation system that tackles limitations with existing nanomanipulation systems. The system utilizes unique in-vacuum, low-power electronics for sensing strain gauge deformation, and flexure based positioner design. The system is capable of producing nanometer resolution closed-loop positioning, subnanometer per minute drift, and friction free, vibration free motion inside SEM. The compact system can be mounted onto most SEM using the standard SEM sample holder, thus allowing the system to be added or removed from an SEM within seconds.

In the applications, the manipulations can be guided by SEM’s real-time vision and handled by operators, or computer generating the motion targets according to the presupposed tasks. Since all the position information is calculated based on the images, this requires the consistency between the image’s and manipulator’s axis. However, there are always install errors, and the two set axes cannot be aligned perfectly. The end-effectors will not match the expected position on images with these errors, and subtle differences will cause large deviation in the nanoscale motion.

There are three key sources of error that demand a full system recalibration prior to use every time. () While handling the system, the force exerted by the human hand causes tiny changes to the mechanical assembly of the system. These changes lead to performance variation of the system in the micronanometer scale. () When the system is installed onto the SEM, there exists small misalignment between the SEM and nanomanipulation system. Even when the installation is carefully conducted, it is impossible to align the system onto SEM stage with nanometer precision. () The internal temperature of the SEM can vary by a few degrees after each setup, which affects the positioner sensor accuracy. All of these factors contribute to inaccuracy in position sensor, which demands recalibration prior to operation each time.

Manually performing the calibrations is time consuming and poor in repeatability. The extended electron beam exposure to the end-effector and sample also leads to significant amount of electron beam induced deposition (EBID) [24]. The rotation of SEM’s image can only align one of the image’s and manipulator’s axes, or This rotation cannot deal with the situation that the manipulator’s axes are not vertical. So an automated calibration process is necessary for quickly measuring the angle and calculating the rotation matrix to align the axes of nanomanipulation robot inside SEM.

This paper developed an automated approach for calibration. The performance of the calibration method is evaluated in terms of speed, calibration repeatability, and positioning accuracy. Compared with our conference paper [25], this paper provides additional details on the error optimization and -axis misalignment compensation; furthermore, more experimental data are presented. The nanomanipulation system is briefly introduced in Section 2. Section 3 presents the automated alignment method based on the SEM image processing. Section 4 analyzed the system’s error transfer and optimization. Finally, Section 5 concludes the paper.

2. System Overview

The nanomanipulation system used for this study consists of four manipulators mounted on a vacuum load-lock compatible carrier [26]. Each manipulator consists of three long range coarse positioners with three high precision fine positioners stacked on top. Coarse positioners are composed of three stick-slip based piezo positioners for positioning. No sensory feedback is implemented in the coarse positioners to minimize heat generation sources. The fine positioners are three flexure guided, preloaded piezo positioners with one strain gauge mounted on each piezo stack. The position sensing principle involves the use of strain gauges mounted on piezo and utilizes time-to-digital convertor (TDC) for strain sensing.

On-board electronics are placed within the aluminum housing of the nanomanipulation system carrier inside the SEM. The external electronics consist of an MCU and arrays of operational amplifiers for driving the piezo stack. After receiving sensor readout from the on-board electronics, the MCU computes the required piezo stack driving voltages based on the PID control law and sends driving voltages to the on-board electronics.

The nanomanipulation system is mounted on a standard SEM sample holder, allowing the system to be added or removed from the SEM stage within seconds.

When manipulating sample inside SEM, the nanomanipulation system is transferred inside SEM through the vacuum load-lock. Then, the sample is transferred inside, too. The axis needs to be aligned to insure the position accuracy. The region of interest can be located manually, and a series of preset operation will be carried out.

Manual calibrations of each fine positioner were conducted under SEM imaging. It involves moving each of the fine positioners back and forth to determine its motion path, followed by raster rotating the SEM image to adjust positioner-image misalignment. The process is completed through manual trial and error. It takes approximately 2 minutes to determine the misalignment angle for one single fine positioner axis. For a 4-manipulator system with 12 axes, the total calibration takes ~24 minutes. After collecting all the misalignment angles between the fine positioners and SEM images, a rotation matrix is used. This calibration process needs to be repeated if the system is removed/added back into the SEM or made physical contact with a human hand. Since the operations are manually performed, the repeatability and accuracy are not stable.

3. Methods

An automated alignment method is proposed in this paper to calculate the rotation matrix. The system moved the end-effector automated and recognized the end-effector’s position based on template matching method. The basic idea is the rotation matrix to transform the expected motion in image coordinate to the motion of every axis:where are the axes’ extension of the manipulator and is the end-effector’s expected motion on the image. Since the -axis’ motion cannot be observed because the SEM’s feedback is 2D image, only - and -axes can be calibrated.

3.1. The Filter and Recognition of Image

High SEM image frame rate is desirable for real-time nanomanipulation, but it leads to degraded image quality. A low accelerating voltage is used to minimize electron induced damage on the sample, but the image signal-to-noise ratio is poor. A filter method is necessary to reduce the image noise real-time.

GPU accelerated nonlocal means (NL-means) method [10, 27] was shown to be effective in reducing SEM image noise in real-time. It is based on globally averaging all the pixels in an image and produces images with lower noise and with more details retained. The algorithm when implemented on graphics processing unit (GPUs) can satisfy the real-time SEM image denoising/processing. Figure 1 is one of the image frames, its denoising result and manually selected template.

After denoising the image, the position of the end-effector can be accurately recognized. In the calibration stage, the background of SEM’s image can be adjusted to almost black, so its difference to the foreground is apparent. Temple matching method is adopted to recognize the position of the end-effector [28]. Considering the noise level of the SEM image, the template matching based on FFT is adopted, which have good performance in rapidity and robustness. As to template , the matching function in image is where is the rotation and expansion of . is the transposition of . The point with maxim cov is the registration point. Figure 2 is the recognition results of stable target (Figure 2(a)) and moving target (Figure 2(b)).

3.2. The Rotation Matrix

The key to calculate the rotation matrix is to obtain the angles between the manipulator’s motion axes and image’s axes. The end-effector was controlled to move a series of positions, followed by image registration and record the position. These serial position coordinates can be fitted and the angles then were obtained. Firstly, define the coordinate system as Figure 3.

As shown in Figure 3, , , and are the axes of image. are the terminate frame of manipulator. , , and are the terminate frame’s projection on plane. , , , and are the angle between and , and , and , and and , respectively.

Specific algorithm is as follows:

() Adjust magnification of SEM to 13,000x. Adjust the SEM imaging parameters to maximize the focus and contrast of the SEM image. Manually select template.

() Control the end-effect to certain position and wait until the error is less than 5 nm.

() Filter the image and reorganize the end-effector’s position .

() Repeat steps () and (), with a serial equidistant position, and record the positions as , .

() Consider is in a straight line to two-dimensional data where both variables are measured with error. Fit the line by Deming Regression and line is ,

() Move it along -axis, and recognize the position automatedly, . Fit them and get the line with slope .

() Then , , and according to the rotation matrix (1), can be calculated asSo we have

In the visual servo, if the position expected on image is , the target of the manipulator’s axis motion is .

3.3. -Axis Misalignment Compensation

As to the manipulator’s -axis, the target image is 2D image according to the SEM’s image-forming principle and cannot reflect the depth information. Many depth predicting methods are developed [12, 29] to gather useful information on -direction. In the application, the movement on -axis often causes the extra movement on the - and -axes due to the install error, which is harmful in operation because it may cause the sample damage while is moved to sample surface. So it needs to be compensated. The new rotation matrix iswhere and .

However, and are both immeasurable, so we firstly moved -axis with a large distance and record the movement on image as and ; then we have , , so the rotation matrix is

In some other occasions, such that the end-effector is far from the sample, the motion of -axis can be embodied in some way to help indicating its motion. When is moving, we can move and a little, so the end-effector seems like moving forward. This mode covers the shortage that cannot be observed directly in SEM’s image. To realize this, we can change the position after alignment:where and is the new rotation matrix.

is very small normally, and the small displacement in - and -direction can indicate the movement of -axis, which allow the operator to clearly sense it and improve the control accuracy. Switching and in different occasion can make the operation conveniently.

4. Error Analysis

For nanoscale motion, small error on positioner axis may cause large difference in the end. Since tolerance for position error is stringent during the data collection process, the method for reducing the error in axis alignment needs to be researched. The alignment method in this paper is to utilize the target registration and slope calculation. The random error mainly comes from the anamorphose, image noise, recognition error, fitting error, and shape variation of end-effector (due to EBID). For the purpose of reducing positioning error, several means can be adopted, such as unbiased filter and recognition algorithm, the independent variable choice in fitting to improve the precision of the slope.

Besides, multiple alignments can also reduce the error. The SEM’s image is different to the physical object and the resulting error can be reduced by iterative method. Realign the axes after the rotation matrix is used in the manipulator system, and multiply the two rotation matrixes; we can get a new matrix. Repeat this process until the error is lower than a certain threshold. On the other hand, the image noise, the error on recognition, and fitting can be reduced by making this same measurement multiple times and taking the average. These two methods both can improve the precision. However, the process of alignment has to be as quick as it can to reduce the EBID’s effects on end-effector. If one time alignment cannot satisfy the required precision, the alignment method needs to be further optimized. So the error’s propagation in this algorithm is researched to compare iteration method and average method under the condition of limited times. The confidence interval of position is adopted to indicate the error.

4.1. The Confidence Interval of Position

Denote the error of the end-effector as , where . The error comes from the rotation matrix , which is obtained by a series of transformation. Without loss of generality, we can analyze the error’s propagation of the unit displacement of all axes (). The main error sources of are the two parameters and . and ’s contribution on final result are too small. Denote the error of and as and ; then we have

Considering the experiments of - and -axes are independent, their errors are independent, too. So the variance is where and are the variance of and , and they can be calculated by jackknife estimate, which is a resampling technique especially useful for variance and bias estimation. The basic idea is to estimate the parameter for each subsample omitting the th observation to estimate the previously unknown value of by average. The variance of iswhere and is sample size.

Since and are symmetric, we calculate -axis firstly. According to the Deming Regression, the parameter ’s confidence interval is , where is confidence level, and is standard error of . The variance of can be got in the same way. So, the confidence interval of is

4.2. The Error Optimizing

Equation (12) is too complicated for analysis, so it should be simplified. In the application, the angle between axes of image and manipulator is very small, that is, , and using second-order Taylor expansion, we can get and . After dropping higher-order infinitesimal, we have

It can be summarized that the confidence interval of end-effector’s position is determined by and . When and are small enough, . So obviously, decreasing and improves the precision of . As to the method in this paper, multiple times alignment and taking the average of and is the optimized choice. When and are not small enough, such that the manipulator was installed incorrectly or it needs to be rotated, the coefficients of and in propagation are greater than 1 significantly, and the system needs to be iterated for a few times until and are small enough and then optimize it with average method.

5. The Experiments

To minimize the thermally induced drift of our system within the vacuum environment, the nanomanipulation system was installed into the SEM for 3 hours prior to the experiment to ensure that the system had reached thermal equilibrium within the SEM. A magnification of 13,000x is selected, which has a field of view (FOV) of 9.7 um × 7.3 um, and each pixel is 12.2 nm in size. Selecting a high image magnification can improve image resolution, but only a smaller nanopositioner motion can be observed within the FOV. The method described in Section 3 is implemented to align the -axis, and the result of - and -axis is shown in Figure 4.

In this experiment, the -axis of manipulator has relatively large difference to the image, and the -axis is smaller. The slopes of these two lines are and , which mean and

As to -axis, only its effect on - and -axis can be observed, and we adjust the - and -axis’s motion according to (7). -axis is moved in larger range, and the displacement on - and -direction is recoded to calculate to compensate - and -axis’s motion. The result is shown in Figure 5. The -axis’s motion range is 3 um, and the effect on - and -axis is −42.51 nm and −21.25 nm.

And the rotation matrix is

After this alignment, the movement of - and -axis is shown in Figure 6.

Please refer to the Supplementary Video for the details (see Supplementary Material available online at The motion of manipulator fits the image’s axis very well after the automated alignment. The whole process takes 90 seconds, and the slope was and , meaning that the error of angle is less than 0.1°.

6. Conclusion

This paper discussed the automated axis alignment method for a manipulator system inside SEM. An end-effector recognition and Deming Regression were adopted to calculate the angle deviation and multiple rotation matrixes are proposed to correct the motion in different condition. The error propagation of this method is analyzed. The average method and iteration method are compared to find out a faster method when more than one time alignment is necessary. The method can speed up the process of alignment and avoid the damage on end-effect due to EBID.

Conflicts of Interest

The authors declare that they have no conflicts of interest.


This work was supported in part by the National Natural Science Foundation of China (61528304, 71401189, and 61473295) and by Beijing Natural Science Foundation (4152054).

Supplementary Materials

This is an automated axis alignment process.

  1. Supplementary Material


  1. C. Shi, K. D. Luu, Q. Yang et al., “Recent advances in nanorobotic manipulation inside scanning electron microscopes,” Microsystems & Nanoengineering, vol. 2, 16024, 2016. View at: Google Scholar
  2. L. Dong, K. Shou, D. R. Frutiger et al., “Engineering multiwalled carbon nanotubes inside a transmission electron microscope using nanorobotic manipulation,” IEEE Transactions on Nanotechnology, vol. 7, no. 4, pp. 508–517, 2008. View at: Publisher Site | Google Scholar
  3. T. Fukuda, F. Arai, and L. Dong, “Assembly of nanodevices with carbon nanotubes through nanorobotic manipulations,” Proceedings of the IEEE, vol. 91, no. 11, pp. 1803–1818, 2003. View at: Publisher Site | Google Scholar
  4. D. J. Bell, L. Dong, B. J. Nelson, M. Golling, L. Zhang, and D. Grützmacher, “Fabrication and characterization of three-dimensional InGaAs/GaAs nanosprings,” Nano Letters, vol. 6, no. 4, pp. 725–729, 2006. View at: Publisher Site | Google Scholar
  5. Y. Zhu, F. Xu, G. Qin, W. Y. Fung, and W. Lu, “Mechanical properties of vapor-Liquid-Solid synthesized silicon nanowires,” Nano Letters, vol. 9, no. 11, pp. 3934–3939, 2009. View at: Publisher Site | Google Scholar
  6. H. D. Espinosa, Y. Zhu, and N. Moldovan, “Design and operation of a MEMS-based material testing system for nanomechanical characterization,” Journal of Microelectromechanical Systems, vol. 16, no. 5, pp. 1219–1231, 2007. View at: Publisher Site | Google Scholar
  7. G. Hwang, H. Hashimoto, D. J. Bell, L. Dong, B. J. Nelson, and S. Schön, “Piezoresistive InGaAs/GaAs nanosprings with metal connectors,” Nano Letters, vol. 9, no. 2, pp. 554–561, 2009. View at: Publisher Site | Google Scholar
  8. A. Lugstein, M. Steinmair, A. Steiger, H. Kosina, and E. Bertagnolli, “Anomalous piezoresistance effect in ultrastrained silicon nanowires,” Nano Letters, vol. 10, no. 8, pp. 3204–3208, 2010. View at: Publisher Site | Google Scholar
  9. Y. Shen, M. Nakajima, Z. Zhang, and T. Fukuda, “Dynamic force characterization microscopy based on integrated nanorobotic AFM and SEM system for detachment process study,” IEEE/ASME Transactions on Mechatronics, vol. 20, no. 6, pp. 3009–3017, 2015. View at: Publisher Site | Google Scholar
  10. Z. Gong, B. K. Chen, J. Liu, and Y. Sun, “Robotic probing of nanostructures inside scanning electron microscopy,” IEEE Transactions on Robotics, vol. 30, no. 3, pp. 758–765, 2014. View at: Publisher Site | Google Scholar
  11. C. Ru, Y. Zhang, Y. Sun et al., “Automated four-point probe measurement of individual nanowires inside a scanning electron microscope,” in Proceedings of the 2010 IEEE International Conference on Automation Science and Engineering (CASE '10), vol. 10, pp. 674–681, IEEE, Toronto, Canada, August 2010. View at: Publisher Site | Google Scholar
  12. J. Li, Y. Zhang, S. To, L. You, and Y. Sun, “Effect of nanowire number, diameter, and doping density on nano-FET biosensor sensitivity,” ACS Nano, vol. 5, no. 8, pp. 6661–6668, 2011. View at: Publisher Site | Google Scholar
  13. K. Aoki, H. T. Miyazaki, H. Hirayama et al., “Microassembly of semiconductor three-dimensional photonic crystals,” Nature Materials, vol. 2, no. 2, pp. 117–121, 2003. View at: Publisher Site | Google Scholar
  14. H.-Y. Chen, C.-L. He, C.-Y. Wang et al., “Far-field optical imaging of a linear array of coupled gold nanocubes: direct visualization of dark plasmon propagating modes,” ACS Nano, vol. 5, no. 10, pp. 8223–8229, 2011. View at: Publisher Site | Google Scholar
  15. M. R. Ahmad, M. Nakajima, M. Kojima, S. Kojima, M. Homma, and T. Fukuda, “Nanofork for single cells adhesion measurement via ESEM-nanomanipulator system,” IEEE Transactions on Nanobioscience, vol. 11, no. 1, pp. 70–78, 2012. View at: Publisher Site | Google Scholar
  16. Z. Gong, B. K. Chen, J. Liu et al., “Fluorescence and SEM correlative microscopy for nanomanipulation of sub-cellular structures,” Light: Science & Applications, vol. 3, no. e224, 2014. View at: Google Scholar
  17. B. K. Chen, D. Anchel, Z. Gong et al., “Nano-dissection and sequencing of DNA at single sub-nuclear structures,” Small, vol. 10, no. 16, pp. 3267–3274, 2014. View at: Publisher Site | Google Scholar
  18. Y. Shen, M. Nakajima, Z. Yang et al., “Single cell stiffness measurement at various humidity conditions by nanomanipulation of a nano-needle,” Nanotechnology, vol. 24, no. 14, 145703, 2013. View at: Publisher Site | Google Scholar
  19. M.-F. Yu, O. Lourie, M. J. Dyer, K. Moloni, T. F. Kelly, and R. S. Ruoff, “Strength and breaking mechanism of multiwalled carbon nanotubes under tensile load,” Science, vol. 287, no. 5453, pp. 637–640, 2000. View at: Publisher Site | Google Scholar
  20. L. Dong, F. Arai, and T. Fukuda, “Destructive constructions of nanostructures with carbon nanotubes through nanorobotic manipulation,” IEEE/ASME Transactions on Mechatronics, vol. 9, no. 2, pp. 350–357, 2004. View at: Publisher Site | Google Scholar
  21. E. C. Heeres, A. J. Katan, M. H. Van Es et al., “A compact multipurpose nanomanipulator for use inside a scanning electron microscope,” Review of Scientific Instruments, vol. 81, no. 2, 023704, 2010. View at: Publisher Site | Google Scholar
  22. S. Fatikow, T. Wich, H. Hülsen, T. Sievers, and M. Jähnisch, “Microrobot system for automatic nanohandling inside a scanning electron microscope,” IEEE/ASME Transactions on Mechatronics, vol. 12, no. 3, pp. 244–252, 2007. View at: Publisher Site | Google Scholar
  23. D. Zhang, J.-M. Breguet, R. Clavel, V. Sivakov, S. Christiansen, and J. Michler, “In situ electron microscopy mechanical testing of silicon nanowires using electrostatically actuated tensile stages,” Journal of Microelectromechanical Systems, vol. 19, no. 3, pp. 663–674, 2010. View at: Publisher Site | Google Scholar
  24. L. Ven Kouwen, A. Botman, and C. W. Hagen, “Focused electron-beam-induced deposition of 3 nm dots in a scanning electron microscope,” Nano Letters, vol. 9, no. 5, pp. 2149–2152, 2009. View at: Publisher Site | Google Scholar
  25. C. Zhou, Z. Wu, L. Deng, Y. Wang, Z. Cao, and M. Tan, “Automated axis alignment for a nano manipulator inside SEM,” in Proceedings of the 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO '16), pp. 1293–1297, IEEE, Qingdao, China, December 2016. View at: Publisher Site | Google Scholar
  26. C. Zhou, Z. Gong, B. K. Chen et al., “A closed-loop controlled nanomanipulation system for probing nanostructures inside scanning electron microscopes,” IEEE/ASME Transactions on Mechatronics, vol. 21, no. 3, pp. 1233–1241, 2016. View at: Publisher Site | Google Scholar
  27. A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, pp. 60–65, usa, June 2005. View at: Publisher Site | Google Scholar
  28. V. Eichhorn, S. Fatikow, T. Wortmann et al., “NanoLab: a nanorobotic system for automated pick-and-place handling and characterization of CNTs,” in Proceedings of the 2009 IEEE International Conference on Robotics and Automation, ICRA '09, pp. 1826–1831, jpn, May 2009. View at: Publisher Site | Google Scholar
  29. V. Eichhorn, S. Fatikow, T. Wich et al., “Depth-detection methods for microgripper based CNT manipulation in a scanning electron microscope,” Journal of Micro-Nano Mechatronics, vol. 4, no. 1-2, pp. 27–36, 2008. View at: Publisher Site | Google Scholar

Copyright © 2017 Chao Zhou et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

More related articles

 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder

Related articles

We are committed to sharing findings related to COVID-19 as quickly as possible. We will be providing unlimited waivers of publication charges for accepted research articles as well as case reports and case series related to COVID-19. Review articles are excluded from this waiver policy. Sign up here as a reviewer to help fast-track new submissions.