A Novel SFP Extracting/Tracking Framework for Rigid Microstructure Measurement

Abstract

3D measurement and reconstruction of rigid microstructures rely on the precise matches of structural feature points (SFPs) between microscopic images. However, most of the existing algorithms fail in extracting and tracking at microscale due to the poor quality of the microscopic images. This paper presents a novel framework for extracting and matching SFPs in microscopic images under two stereo microscopic imaging modes in our system, that is, fixed-positioning stereo and continuous-rotational stereo modes, respectively. A 4-DOF (degree of freedom) micro visual measurement system is developed for 3D projective structural measurement of rigid microstructures using the SFPs obtained from microscopic images by the proposed framework. Under the fixed-positioning stereo mode, a similarity-pictorial structure algorithm is designed to preserve the illumination invariance in SFPs matching, while a method based on particle filter with affine transformation is developed for accurate tracking of multiple SFPs in image sequence under the continuous-rotational stereo mode. The experimental results demonstrate that the problems of visual distortion, illumination variability, and irregular motion estimation in micro visual measurement process can be effectively resolved by the proposed framework.

1. Introduction

Micro stereovision technology makes it possible to achieve 2D/3D information extraction and 3D reconstruction. It has been extensively used in micromanipulation, microassembly, microrobot navigation and bioengineering, and so forth. 3D reconstruction of the small objects in microscopic image is a challenging work according to the small sizes. Existing methods as structured-light-based 3D reconstruction [1, 2], and binocular stereo methods [3]. are not suitable directly for micro applications and then definitely need to be modified. For rigid 3D microstructures, the accurate matching of structural feature points between microscopic images is one of the most important steps in micro stereovision computation. In this paper we define the structural feature points (SFPs) as those key points that can fully represent the rigid microstructures, which are those intersection points of 3 planes on the micro 3D structure (Figure 1(a)). Although the vision method based on tracking of feature points plays an important role in normal-scale computer vision applications as 3D reconstruction, segmentation, and recognition, there is little research focused on the microscale applications except for medical applications. This is due to poor contrast and worse quality of microscopic image mainly resulting from the imaging complexity of optical microscope. The complex structure of microscopic lens system [4] requires the variety of optical elements that can introduce a wide array of image distortions.

(a) SFPs

(b) Harris

(c) Harris-affine

(d) SIFT

Figure 1 

Structural feature points on a rigid 3D microstructure (a) and the failed detecting and matching results of some popular methods (b), (c), and (d). These results failed to locate precisely on the positions of SFPs.

In this paper we present a framework for extracting and matching SFPs in microscopic images for dealing with fixed-positioning and continuous-rotational stereo imaging modes, respectively. We believe that the proposed framework perfectly plays an important role in 3D reconstruction based on microscopic image.

First, since there exist some specific drawbacks in microscopic images as blurred edges, geometrical distortions, serious dispersion, and disturbance by noises (especially coming from illumination changes), which result in more troubles for SFPs’ detection and matching, many popular key point detecting and matching methods [5–9] failed to achieve similar accuracy in microscopic images (Figures 1(b)–1(d)) in comparison with those results obtained when employed on normal-scale images. New methods that are suitable for SFPs’ extracting and matching in microscopic images are highly required. We develop a novel illumination-invariant method based on similarity-pictorial structure algorithm (similarity-PS) to solve the SFPs’ extracting and matching problem in the fixed-positioning stereo mode.

Second, existing feature point tracking algorithms for continuous image sequences are usually classified as methods based on template, motion parameters, and color patch [10], none of them meet the needs of SFPs’ tracking in microscale images. The representative research achievements include: continuously adaptive mean shift (Camshift) [11] taking color histogram as object mode in rich-color images for the tracking tasks, which performs poorly when tracking the feature points in complicated background with areas of similar color. Yao and Chellappa first designed a probabilistic data association filter with extended Kalman filter (EKF) [12] to estimate the rotational motion and continuously track the feature points in frames, which effectively resolved occlusion problem. However, the real location must be predicted by probability analysis during arbitrary moving process of the object.

Buchanan proposed a combining local and global motion models with Kanade-Lucas-Tomasi (KLT) tracker to accurately track multiple feature points of nonrigid moving objects [13]. But if motion predictions cannot be made for the subsequences consisting of the initial frames, this strategy must fail. Kwon et al. presented a template tracking approach based on applying the particle filtering (PF) algorithm on the affine group [14, 15], which can accurately track a large single object, but it performs well only in single template tracking. We extended their method to the multiple-points tracking case in the proposed monocular microvision system in the continuous-rotational stereo mode. Since the images of the measuring rigid micro structure keep fixed spatial relationship of global structures and local features affine invariabilities during the rotating process, the tracking problem is greatly simplified with affine transformation and covariance descriptor in our framework.

2. Proposed Framework

2.1. Imaging Modes of Proposed Micro Stereoscopic Measurement System

We developed a 4-DOF micro stereoscopic measurement system for 3D micro objects measurement. The system consists of a 4-DOF stage, a fixedmounted illumination and a SLM-based optical imaging system. A monocular tube microscope is used in the system for reducing the imaging complexity. In our system the stereoscopic imaging relationship can be realized in two modes.

(i) Fixed-Positioning Stereo Mode (Mode 1)
The rotational transform by a fixed rotation angle is performed, followed by a tilting motion. Images are captured at the end of each movement. Problem: The changes of illumination direction bring huge contrast and intensity changes to the microscopic images captured at different position.

(ii) Continuous-Rotational Stereo Mode (Mode 2)
Tilting movement is performed firstly, followed by a continuously rotational movement. Image sequences are captured during the rotation process. Problem:The motion blur caused by continuous rotational motion will decrease the quality of microscopic image sequences.

2.2. Similarity-PS Method for SFPs Matching of Mode 1

2.2.1. Pictorial Structure Method

A PS method is represented by a collection of parts which has spatial relationships between certain pairs. The PS model can be expressed by a graph 𝐺=(𝑉,𝐸), where the vertices 𝑉={𝑣₁,𝑣₂,𝑣₃,…,𝑣_𝑛} correspond to the parts and {𝑣_𝑖,𝑣_𝑗}∈𝐸 present the edge for each pair of connected parts 𝑣_𝑖 and 𝑣_𝑗. An object is expressed by a configuration 𝐿=(𝑙₁,𝑙₂…,𝑙_𝑛) where 𝑙_𝑖 represents the location for each part 𝑣_𝑖. For each part 𝑣_𝑖, the appearance match cost function 𝑎_𝑖(𝐼,𝑙) represents how well the part matches the image 𝐼 when placed at location 𝑙. A simple pixel template matching is used for this cost function in [16]. The connections between the locations of parts present the structure match cost. The cost function 𝑡_𝑖𝑗(𝑣𝑖,𝑣𝑗) represents how well the locations 𝑙_𝑖 of 𝑣_𝑖 and 𝑙_𝑗 of 𝑣_𝑗 agree with the object model. Therefore, the cost function 𝐿^∗ for PS includes 2 parts (the appearance cost function and structure cost function). 𝐿^∗=argmin⎛⎜⎜⎝_{𝑣𝑖∈𝑉}𝑎_𝑖1,𝑙_𝑗+_{𝑣𝑖,𝑣𝑗∈𝐸}𝑡_𝑖𝑗𝑙_𝑖,𝑙_𝑗⎞⎟⎟⎠.(2.1) The best match of SFPs can be obtained by minimizing 𝐿^∗.

2.2.2. The Local Self-Similarity Descriptor

Self-similarity descriptor is proposed by Buchanan and Fitzgibbon [13].

Figure 2 illustrates the procedure for generating the self-similarity descriptor 𝑑_𝑞 centered at 𝑞 with a large image. 𝑞 is a pixel in the input image.

Figure 2 

Extract the local self-similarity.

The green square region is a small image patch (typically5∗5,3∗3) centered at 𝑞. The lager blue square region is a big image region (typically 30∗30,40∗40) centered with 𝑞, too. First, the small image patch is compared with a larger image region using sum of square differences (SSDs). The CIE 𝐿∗𝑎∗𝑏 color space transforming is needed for color images. Second, the correlation surface is normalized to eliminate illumination influences. Finally, the normalized correlation surface is transformed into a “correlation surface” 𝑆_𝑞(𝑥,𝑦)𝑆_𝑞(𝑥,𝑦)=exp−SSD^𝑞(𝑥,𝑦)maxvar_noise,var_auto(𝑞),(2.2) where SSD_𝑞(𝑥,𝑦) is the normalized correlation surface and var_noise is a constant number that corresponds to acceptable photometric variations (in color, illumination or due to noise, which is 150 in the paper). var_auto(𝑞) is the maximal variance of the difference of all patches within a very small neighborhood of 𝑞 (of radius 1) relative to the patch centered at 𝑞.

The correlation surface SSD_𝑞(𝑥,𝑦) is then transformed into log-polar coordinates centered at 𝑞 and portioned into 20∗4 bins (𝑚=20 angles, 𝑛=4 radius). We choose the maximal value in every bin (it can help the descriptor to adapt to nonrigid deformation). We choose all the maximal values to form an 𝑚∗𝑛 vector as a self-similarity descriptor centered at 𝑞. Finally, this descriptor vector is normalized to the range [0⋯1] by linearly stretching its values.

2.2.3. Similarity-PS Algorithm

We introduce the local “self-similarity” descriptor-matching approach for SFPs’ detection into a simplified PS model. This subsection covers 3 main steps as follows

Extraction of the “Template” Description T
Manually marked SFPs’ coordinates on the micro structure are used for the training process. We describe the marked points by self-similarity descriptors. For every point, the average values of all the examples of descriptor as a trained descriptor are calculated, acting as the appearance description for the model.

Dense Computation of the Local Self-Similarity Descriptor
These descriptors 𝑑_𝑞 are computed throughout the tested image I with 2 pixels apart from each other in this paper. The higher precision will be obtained if the searching process covers every pixel.

Detection of Similar Descriptors of T within I
The region centered at SFPs is chosen as the interesting region in 𝐼, which has the smallest weighted Euclidean distance between self-similarity descriptor vector within 𝐼 and the one from the training descriptor. The coordinates of all interest points in 𝐼 are recorded as the locations for the candidate key points. We usually find out many candidate key points for one marked point. In our experiment there are 200 points being chosen (but it varies for different templates), and their Euclidean distances between the candidate key points descriptors and the trained descriptors are recorded as 𝑎_𝑖(𝐼,𝑙_𝑗) which are linearly normalized to the range of [0⋯1]. Therefore the appearance cost function 𝑎_𝑖(𝐼,𝑙_𝑗) is determined. We substituted the appearance model in (2.1) with the obtained self-similarity model; then, the best-matching SFPs for the PS model can be obtained by minimizing 𝐿^∗.
Since the microscopic images are not always of the same size, to satisfy the scale invariance, we calculate the self-similarity descriptors at multiple scales both on the patterns and testing images. Moreover, the scale-invariant structures are obtained by multiplying a scale factor to all the mean and variance value of structural distances of SFPs.

2.3. PF-Based Multiple SFPs Tracking of Mode 2

2.3.1. The Affine Motion of Tracking Points

A single tracked point 𝑏^𝑡(𝑖)=(𝑥^𝑡(𝑖),𝑦^𝑡(𝑖))^𝑇(𝑖=1,…,𝑘,𝑡=1,…,𝑁, representing the number of tracked points and frames, resp.) cannot satisfy the accurate tracking in micro sequences

therefore it is necessary to take advantage of templates. Initializing multiple points with a set of given center coordinates 𝑏_𝑡=(𝑥^𝑡(1),𝑦^𝑡(1),…,𝑥^𝑡(𝑖),𝑦^𝑡(𝑖))^𝑇, a set of small regions point templates 𝑇(𝑋^𝑡(𝑖)) are denoted as a group of windows which surround each tracked point for describing their main characteristics. The coordinates of each 𝑇(𝑋^𝑡(𝑖)) are obtained via multiplication, the homogeneous coordinates with affine transformation matrix 𝑋^𝑡(𝑖),so the most important step in tracking process is to calculate the state 𝑋^𝑡(𝑖), for all point templates in each frame. An affine motion can be divided into a translation 𝑀^𝑡(𝑖) and a rotation 𝑅_𝑡 all the points of a rigid object are rotating with same angular velocity. The translation vectors are denoted by 𝑀^𝑡(𝑖)=(𝑀𝑥^𝑡(1),𝑀𝑦^𝑡(1),…,𝑀𝑥^𝑡(𝑖),𝑀𝑦^𝑡(𝑖))^𝑇, and the affine motion action of a point can be written as ℓ(𝑏^𝑡(𝑖))=𝑅_𝑡𝑏^𝑡(𝑖)+𝑀^𝑡(𝑖) and 𝑅^𝑡𝑀^𝑡(𝑖)01=expΓ0𝛾0,(2.3) where 𝑙 is a linear function representing the affine motion of points 𝑏^𝑡(𝑖), 𝑅_𝑡 is an invertible2×2rotation matrix, and 𝑅_𝑡∈𝑅², 𝑀^𝑡(𝑖) is a 2×𝑖 translation vector 𝑀^𝑡(𝑖)∈𝑅². The second form of (2.1) using the exponential map is for the convenience of denoting the affine transformation of the imaging process.

2.3.2. Tracking by Particle Filter with Affine Motion

State estimate and sample for tracked points in each frame: the efficiency of the particle filter tracking algorithm mainly relies on the importance of random sampling and calculate the weights 𝑤^{𝑡(𝑖,𝑗)}. The measurement likelihood 𝑝(𝑦^𝑡(𝑖)𝑋^𝑡(𝑖)) is independent of the current state particles 𝑋^{(𝑖,𝑗)0∶𝑡}. The measurement state equation can be expressed in the discrete setting as 𝑦𝑋^𝑡(𝑖)=𝑓𝑋^{(𝑖)𝑡−1}exp𝑎_𝑖log𝑋^{(𝑖)𝑡−2}⁻¹𝑋^{(𝑖)𝑡−1}+³_𝑚=1𝜉_𝑚𝜀𝜏_𝑡+𝑣^𝑡,(2.4) where v_t is a measurement zero-mean Gaussian noise. 𝑎_𝑖 is the AR process parameter. τ_t is zero-mean Gaussian noise. 𝜀=(1/12)^1/2 represents obtaining 12 frames per second. The affine transformation basis elements 𝜉_𝑚(𝑚=1,…,3) denoting the templates have produced translation, shearing or scaling transformation.

It is necessary to approximate the particles {𝑅^𝑡(𝑖,1),…,𝑅^{𝑡(𝑖,𝐺)}} and resample according to their weights at every timestep. To optimize the computational procedure and avoid directly calculating, all the resample particles are expected to be quite similar to each other. Denote 𝑀^𝑡(𝑖)as the arithmetic mean of 𝑀^{𝑡(𝑖,𝑗)}, the sample mean of 𝑋^{𝑡(𝑖,𝑗)} can be approximated as 𝑅_𝑡(𝑖)𝑀_𝑡(𝑖)01=expΓ0𝛾0=⎡⎢⎢⎢⎣𝑅_{𝑡,max(𝑖)}exp1𝐺^𝐺_𝑗=1log𝑅_{𝑡,max(𝑖)−1}𝑅^{𝑡(𝑖,𝑗)}𝑀^𝑡(𝑖)01⎤⎥⎥⎥⎦.(2.5)Calculate covariance with descriptor for the tracked point templates: the spatial structure constraint vector ℎ=(𝑥^𝑡(𝑖),𝑦^𝑡(𝑖)𝑏^{𝑡(𝑖,𝑑)},𝑏^{𝑡(𝑖,𝜃)},𝐼(𝑋^𝑡(𝑖)),𝐼_𝑥,𝐼_𝑦,tan⁻¹(𝐼_𝑥/𝐼_𝑦),𝐼_𝑥𝑥,𝐼_𝑦𝑦)^𝑇 is added in our method for minimizing the influence of illumination, distortion, motion-blur, and noise interference. Here 𝑏^{𝑡(𝑖,𝑑)},𝑏^{𝑡(𝑖,𝜃)} represent the distance and the angle between origin (or polar axis) and each tracked pixel in polar coordinate system. 𝐼(𝑋^𝑡(𝑖)) denotes the image pixel intensity. 𝐼_𝑥, 𝐼_𝑦, 𝐼_𝑥𝑥, 𝐼_𝑦𝑦 represent the first- and second-order image derivatives in the Cartesian coordinates system [14, 16]. 𝑠 is the size of template window ℎ is the mean value of ℎ(𝑋^𝑡(𝑖)). The covariance descriptor 𝑆 of the point templates patches can be given as 𝑆_𝑋^(𝑖)𝑡=0=1𝑠−1^𝑠_𝑝=1ℎ^{(𝑋𝑡(𝑖))𝑝}−ℎℎ_(𝑋^𝑡(𝑖)_)𝑝−ℎ^𝑇.(2.6)Measure relative distance: for ensuring the covariance descriptors are changing successively, it is necessary to collect image covariance descriptors and calculate the principal eigenvector and the geodesic distance between two group elements {𝑆(𝑋^𝑡(𝑖)),𝑆}and{𝐼(𝑋^𝑡(𝑖)),𝐼}. The measurement function is defined using the distance-from-feature space, distance-in-feature space, and similarity comparison purposes [18] then the measurement equation in [19] can be more explicitly expressed as 𝑦^𝑡(𝑖)=⎡⎢⎢⎢⎢⎢⎢⎢⎣⎷‖‖log𝑆_{𝑋𝑡(𝑖)}−log𝑆‖‖²−^𝑀_𝑛=1𝑐_𝑛2⎷_𝑀_𝑛=1𝑐^2𝑛𝜌_𝑛‖‖𝐼_{𝑋𝑡(𝑖)}−𝐼‖‖⎤⎥⎥⎥⎥⎥⎥⎥⎦+𝑣^𝑡(𝑖),(2.7) where c_n is the projection coefficient and ρ_n is the eigenvalues corresponding to principal eigenvector, and the two parameters are used to calculate the distance-in-feature space, and 𝐼 represents the point mean intensity. The measurement likelihood is described as 𝑝𝑦^𝑡(𝑖)∣𝑋^𝑡(𝑖)∝exp𝑦^{𝑡𝑇(𝑖)}𝑅⁻¹𝑦^𝑡(𝑖),(2.8) where 𝑅 is the covariance of zero-mean Gaussian noise 𝑣_𝑡. When the measurement likelihood 𝑝(𝑦^𝑡(𝑖)∣𝑋^𝑡(𝑖)) is gained, we can calculate and normalize the importance weights for the tracked points and then realize multiple SFPs’ tracking in long microscopic sequences.

3. Experimental Results and Discussion

The SFPs’ detecting/matching experiments are implemented on a standard microchip with 3D microstructures. Some of the results by both PS algorithm and proposed method are shown in Figure 3, while the contrast accumulated error measured by the Euclidean distance in pixels of the tracking points to the ground truth positions is shown in Figure 4. It proves that our method enhances the correct detecting rate of SFPs’ for images under large changed illumination in Mode 1. The reason for this improvement is that the changed gray value is largely caused by the varying illumination while the local structure keeps static. Therefore, it is a good choice to introduce the local self-similarity descriptors into PS for our application.

Figure 3 

Tracking results in Mode 1 by PS algorithm (left two images) and our proposed method (right two images). Each image pairs are corresponding to different illumination conditions.

Figure 4 

Comparing the accumulated error distribution curves of PS method and Similarity-PS method in the Mode 1. Points number from 0 to 250 are tracking points on the images which are captured under different illumination condition with the rest ones.

We also provide some SFPs’ tracking results in Mode 2 by our proposed method and KLT for comparison, as shown in Figure 5. The tracking results by the proposed method obviously show a higher localization accuracy of the SFPs’ in long micro image sequence.

Figure 5 

The first and second columns show the accurate tracking results of proposed PF with affine transformation, while the third and fourth columns demonstrate the tracking results of KLT algorithm with lost or dislocated tracks.

For further demonstrating the effectiveness of our method, we present some of the 3D projective reconstruction results based on the tracking results in Figure 6. Obviously the results of the proposed method perfectly reconstruct the 3D structure from the microscopic sequences, while the KLT method failed.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6 

3D projective reconstruction results by using multiple-point tracking results by PF with affine transformation (the first row) and multiple-point tracking results by KLT algorithm (the second row) respectively. The irregular red curves represent the camera relative moving trajectory. Obviously our method outperforms KLT greatly in the application of 3D reconstruction from microscopic sequences.

4. Conclusions

This paper developed a framework of SFPs’ extracting and matching in microscopic images for 3D micro stereoscopic measurement in two stereo imaging modes. The proposed SFPs’ tracking framework ensures the illumination invariance and the robustness in the fixed-positioning stereo mode and continuous-rotational stereo mode, respectively. The effectiveness of our tracking framework has been empirically verified in visual 3D projective reconstruction with microscopic images. In Mode 2 of our system, there is an inevitable tracking error caused by motion blur. Therefore we plan to use the method described in [20] to deal with this problem in our future work. Our future research will also focus on the micro visual measurement planning (similar to the method described in [21]) and optimal 3D micro model representation [22].

Acknowledgments

This work was supported by the National Natural Science Foundation of China (NSFC-60870002, 60802087, 60573123, 60605013), NCET, and the Science and Technology Department. of Zhejiang Province (2009C21008, 2010R10006, Y1090592, Y1110688).