Abstract
In the motion of probing nanostructures, repeating position and movement is frequently happing and tolerance for position error is stringent. The consistency between the axis of manipulators and image is very significant since the visual servo is the most important tool in the automated manipulation. This paper proposed an automated axis alignment method for a nanomanipulator inside the SEM by recognizing the position of a closed-loop controlling the end-effector, which can characterize the relationship of these two axes, and then the rotation matrix can be calculated accordingly. The error of this method and its transfer function are also calculated to compare the iteration method and average method. The method in this paper can accelerate the process of axis alignment to avoid the electron beam induced deposition effect on the end tips. Experiment demonstration shows that it can achieve a 0.1-degree precision in 90 seconds.
1. Introduction
Nanoscale manipulations inside scanning electron microscope (SEM) have found many uses in various fields [1]. Examples include nanomaterial characterizations [2–9], nanoelectronic probing [10–12], nanodevices prototyping [13], photonics [14, 15], and biological researches [16–18]. The most common setup involves installing piezo-based manipulator into an SEM. A joystick is used to control the motion of the manipulator, and the SEM provides the real-time image feedback. This combination provides an intuitive hand-eye coordinated method to interact with objectives and in micrometer and nanometer scale.
Several SEM based nanomanipulation systems have been reported in the literature [19–23]. There are also commercially available systems from Kleindiek, DCG Systems (Previously Zyvex), SmarAct, Klocke, and Attocube. The majority of these systems do not have position sensor integrated, thus repeatable motions cannot be made, and the manipulation efficiency relies heavily on the skill of the human operator. Other systems contain optical encoders for position feedback, but the heat generated by the laser diode is difficult to dissipate inside vacuum, leading to high position drift rate. The use of mechanical sliding rails for guiding the piezo stick-slip motion is not repeatable due to frictions in the interfaces and the deformation in the mechanical rails. The stick-slip motion also creates mechanical vibration while in motion, which can cause end-effector (the device at the end of a robotic arm, such as claws and needles) or sample damage.
We have previously reported a new load-lock compatible nanomanipulation system that tackles limitations with existing nanomanipulation systems. The system utilizes unique in-vacuum, low-power electronics for sensing strain gauge deformation, and flexure based positioner design. The system is capable of producing nanometer resolution closed-loop positioning, subnanometer per minute drift, and friction free, vibration free motion inside SEM. The compact system can be mounted onto most SEM using the standard SEM sample holder, thus allowing the system to be added or removed from an SEM within seconds.
In the applications, the manipulations can be guided by SEM’s real-time vision and handled by operators, or computer generating the motion targets according to the presupposed tasks. Since all the position information is calculated based on the images, this requires the consistency between the image’s and manipulator’s axis. However, there are always install errors, and the two set axes cannot be aligned perfectly. The end-effectors will not match the expected position on images with these errors, and subtle differences will cause large deviation in the nanoscale motion.
There are three key sources of error that demand a full system recalibration prior to use every time. () While handling the system, the force exerted by the human hand causes tiny changes to the mechanical assembly of the system. These changes lead to performance variation of the system in the micronanometer scale. () When the system is installed onto the SEM, there exists small misalignment between the SEM and nanomanipulation system. Even when the installation is carefully conducted, it is impossible to align the system onto SEM stage with nanometer precision. () The internal temperature of the SEM can vary by a few degrees after each setup, which affects the positioner sensor accuracy. All of these factors contribute to inaccuracy in position sensor, which demands recalibration prior to operation each time.
Manually performing the calibrations is time consuming and poor in repeatability. The extended electron beam exposure to the end-effector and sample also leads to significant amount of electron beam induced deposition (EBID) [24]. The rotation of SEM’s image can only align one of the image’s and manipulator’s axes, or This rotation cannot deal with the situation that the manipulator’s axes are not vertical. So an automated calibration process is necessary for quickly measuring the angle and calculating the rotation matrix to align the axes of nanomanipulation robot inside SEM.
This paper developed an automated approach for calibration. The performance of the calibration method is evaluated in terms of speed, calibration repeatability, and positioning accuracy. Compared with our conference paper [25], this paper provides additional details on the error optimization and -axis misalignment compensation; furthermore, more experimental data are presented. The nanomanipulation system is briefly introduced in Section 2. Section 3 presents the automated alignment method based on the SEM image processing. Section 4 analyzed the system’s error transfer and optimization. Finally, Section 5 concludes the paper.
2. System Overview
The nanomanipulation system used for this study consists of four manipulators mounted on a vacuum load-lock compatible carrier [26]. Each manipulator consists of three long range coarse positioners with three high precision fine positioners stacked on top. Coarse positioners are composed of three stick-slip based piezo positioners for positioning. No sensory feedback is implemented in the coarse positioners to minimize heat generation sources. The fine positioners are three flexure guided, preloaded piezo positioners with one strain gauge mounted on each piezo stack. The position sensing principle involves the use of strain gauges mounted on piezo and utilizes time-to-digital convertor (TDC) for strain sensing.
On-board electronics are placed within the aluminum housing of the nanomanipulation system carrier inside the SEM. The external electronics consist of an MCU and arrays of operational amplifiers for driving the piezo stack. After receiving sensor readout from the on-board electronics, the MCU computes the required piezo stack driving voltages based on the PID control law and sends driving voltages to the on-board electronics.
The nanomanipulation system is mounted on a standard SEM sample holder, allowing the system to be added or removed from the SEM stage within seconds.
When manipulating sample inside SEM, the nanomanipulation system is transferred inside SEM through the vacuum load-lock. Then, the sample is transferred inside, too. The axis needs to be aligned to insure the position accuracy. The region of interest can be located manually, and a series of preset operation will be carried out.
Manual calibrations of each fine positioner were conducted under SEM imaging. It involves moving each of the fine positioners back and forth to determine its motion path, followed by raster rotating the SEM image to adjust positioner-image misalignment. The process is completed through manual trial and error. It takes approximately 2 minutes to determine the misalignment angle for one single fine positioner axis. For a 4-manipulator system with 12 axes, the total calibration takes ~24 minutes. After collecting all the misalignment angles between the fine positioners and SEM images, a rotation matrix is used. This calibration process needs to be repeated if the system is removed/added back into the SEM or made physical contact with a human hand. Since the operations are manually performed, the repeatability and accuracy are not stable.
3. Methods
An automated alignment method is proposed in this paper to calculate the rotation matrix. The system moved the end-effector automated and recognized the end-effector’s position based on template matching method. The basic idea is the rotation matrix to transform the expected motion in image coordinate to the motion of every axis:where are the axes’ extension of the manipulator and is the end-effector’s expected motion on the image. Since the -axis’ motion cannot be observed because the SEM’s feedback is 2D image, only - and -axes can be calibrated.
3.1. The Filter and Recognition of Image
High SEM image frame rate is desirable for real-time nanomanipulation, but it leads to degraded image quality. A low accelerating voltage is used to minimize electron induced damage on the sample, but the image signal-to-noise ratio is poor. A filter method is necessary to reduce the image noise real-time.
GPU accelerated nonlocal means (NL-means) method [10, 27] was shown to be effective in reducing SEM image noise in real-time. It is based on globally averaging all the pixels in an image and produces images with lower noise and with more details retained. The algorithm when implemented on graphics processing unit (GPUs) can satisfy the real-time SEM image denoising/processing. Figure 1 is one of the image frames, its denoising result and manually selected template.
(a)
(b)
(c)
After denoising the image, the position of the end-effector can be accurately recognized. In the calibration stage, the background of SEM’s image can be adjusted to almost black, so its difference to the foreground is apparent. Temple matching method is adopted to recognize the position of the end-effector [28]. Considering the noise level of the SEM image, the template matching based on FFT is adopted, which have good performance in rapidity and robustness. As to template , the matching function in image is where is the rotation and expansion of . is the transposition of . The point with maxim cov is the registration point. Figure 2 is the recognition results of stable target (Figure 2(a)) and moving target (Figure 2(b)).
(a)
(b)
3.2. The Rotation Matrix
The key to calculate the rotation matrix is to obtain the angles between the manipulator’s motion axes and image’s axes. The end-effector was controlled to move a series of positions, followed by image registration and record the position. These serial position coordinates can be fitted and the angles then were obtained. Firstly, define the coordinate system as Figure 3.
As shown in Figure 3, , , and are the axes of image. are the terminate frame of manipulator. , , and are the terminate frame’s projection on plane. , , , and are the angle between and , and , and , and and , respectively.
Specific algorithm is as follows:
() Adjust magnification of SEM to 13,000x. Adjust the SEM imaging parameters to maximize the focus and contrast of the SEM image. Manually select template.
() Control the end-effect to certain position and wait until the error is less than 5 nm.
() Filter the image and reorganize the end-effector’s position .
() Repeat steps () and (), with a serial equidistant position, and record the positions as , .
() Consider is in a straight line to two-dimensional data where both variables are measured with error. Fit the line by Deming Regression and line is ,
() Move it along -axis, and recognize the position automatedly, . Fit them and get the line with slope .
() Then , , and according to the rotation matrix (1), can be calculated asSo we have
In the visual servo, if the position expected on image is , the target of the manipulator’s axis motion is .
3.3. -Axis Misalignment Compensation
As to the manipulator’s -axis, the target image is 2D image according to the SEM’s image-forming principle and cannot reflect the depth information. Many depth predicting methods are developed [12, 29] to gather useful information on -direction. In the application, the movement on -axis often causes the extra movement on the - and -axes due to the install error, which is harmful in operation because it may cause the sample damage while is moved to sample surface. So it needs to be compensated. The new rotation matrix iswhere and .
However, and are both immeasurable, so we firstly moved -axis with a large distance and record the movement on image as and ; then we have , , so the rotation matrix is
In some other occasions, such that the end-effector is far from the sample, the motion of -axis can be embodied in some way to help indicating its motion. When is moving, we can move and a little, so the end-effector seems like moving forward. This mode covers the shortage that cannot be observed directly in SEM’s image. To realize this, we can change the position after alignment:where and is the new rotation matrix.
is very small normally, and the small displacement in - and -direction can indicate the movement of -axis, which allow the operator to clearly sense it and improve the control accuracy. Switching and in different occasion can make the operation conveniently.
4. Error Analysis
For nanoscale motion, small error on positioner axis may cause large difference in the end. Since tolerance for position error is stringent during the data collection process, the method for reducing the error in axis alignment needs to be researched. The alignment method in this paper is to utilize the target registration and slope calculation. The random error mainly comes from the anamorphose, image noise, recognition error, fitting error, and shape variation of end-effector (due to EBID). For the purpose of reducing positioning error, several means can be adopted, such as unbiased filter and recognition algorithm, the independent variable choice in fitting to improve the precision of the slope.
Besides, multiple alignments can also reduce the error. The SEM’s image is different to the physical object and the resulting error can be reduced by iterative method. Realign the axes after the rotation matrix is used in the manipulator system, and multiply the two rotation matrixes; we can get a new matrix. Repeat this process until the error is lower than a certain threshold. On the other hand, the image noise, the error on recognition, and fitting can be reduced by making this same measurement multiple times and taking the average. These two methods both can improve the precision. However, the process of alignment has to be as quick as it can to reduce the EBID’s effects on end-effector. If one time alignment cannot satisfy the required precision, the alignment method needs to be further optimized. So the error’s propagation in this algorithm is researched to compare iteration method and average method under the condition of limited times. The confidence interval of position is adopted to indicate the error.
4.1. The Confidence Interval of Position
Denote the error of the end-effector as , where . The error comes from the rotation matrix , which is obtained by a series of transformation. Without loss of generality, we can analyze the error’s propagation of the unit displacement of all axes (). The main error sources of are the two parameters and . and ’s contribution on final result are too small. Denote the error of and as and ; then we have
Considering the experiments of - and -axes are independent, their errors are independent, too. So the variance is where and are the variance of and , and they can be calculated by jackknife estimate, which is a resampling technique especially useful for variance and bias estimation. The basic idea is to estimate the parameter for each subsample omitting the th observation to estimate the previously unknown value of by average. The variance of iswhere and is sample size.
Since and are symmetric, we calculate -axis firstly. According to the Deming Regression, the parameter ’s confidence interval is , where is confidence level, and is standard error of . The variance of can be got in the same way. So, the confidence interval of is
4.2. The Error Optimizing
Equation (12) is too complicated for analysis, so it should be simplified. In the application, the angle between axes of image and manipulator is very small, that is, , and using second-order Taylor expansion, we can get and . After dropping higher-order infinitesimal, we have
It can be summarized that the confidence interval of end-effector’s position is determined by and . When and are small enough, . So obviously, decreasing and improves the precision of . As to the method in this paper, multiple times alignment and taking the average of and is the optimized choice. When and are not small enough, such that the manipulator was installed incorrectly or it needs to be rotated, the coefficients of and in propagation are greater than 1 significantly, and the system needs to be iterated for a few times until and are small enough and then optimize it with average method.
5. The Experiments
To minimize the thermally induced drift of our system within the vacuum environment, the nanomanipulation system was installed into the SEM for 3 hours prior to the experiment to ensure that the system had reached thermal equilibrium within the SEM. A magnification of 13,000x is selected, which has a field of view (FOV) of 9.7 um × 7.3 um, and each pixel is 12.2 nm in size. Selecting a high image magnification can improve image resolution, but only a smaller nanopositioner motion can be observed within the FOV. The method described in Section 3 is implemented to align the -axis, and the result of - and -axis is shown in Figure 4.
(a)
(b)
In this experiment, the -axis of manipulator has relatively large difference to the image, and the -axis is smaller. The slopes of these two lines are and , which mean and
As to -axis, only its effect on - and -axis can be observed, and we adjust the - and -axis’s motion according to (7). -axis is moved in larger range, and the displacement on - and -direction is recoded to calculate to compensate - and -axis’s motion. The result is shown in Figure 5. The -axis’s motion range is 3 um, and the effect on - and -axis is −42.51 nm and −21.25 nm.
And the rotation matrix is
After this alignment, the movement of - and -axis is shown in Figure 6.
(a)
(b)
Please refer to the Supplementary Video for the details (see Supplementary Material available online at https://doi.org/10.1155/2017/3982503). The motion of manipulator fits the image’s axis very well after the automated alignment. The whole process takes 90 seconds, and the slope was and , meaning that the error of angle is less than 0.1°.
6. Conclusion
This paper discussed the automated axis alignment method for a manipulator system inside SEM. An end-effector recognition and Deming Regression were adopted to calculate the angle deviation and multiple rotation matrixes are proposed to correct the motion in different condition. The error propagation of this method is analyzed. The average method and iteration method are compared to find out a faster method when more than one time alignment is necessary. The method can speed up the process of alignment and avoid the damage on end-effect due to EBID.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (61528304, 71401189, and 61473295) and by Beijing Natural Science Foundation (4152054).
Supplementary Materials
This is an automated axis alignment process.