Robust Evolution Method of Active Contour Models and Application in Segmentation of Image Sequence
Active contour models are widely used in image segmentation. In order to obtain ideal object boundary, researchers utilize various information to define new models for image segmentation. However, the models could not meet all scenes of image. In this paper, we propose a block evolution method to improve the robustness of contour evolution. A block matrix is consisted of contours of former iterations and contours of shape prior, and a nuclear norm of the matrix is a measure of the similarity of these shapes. The constraint of the nuclear norm minimization is imposed on the evolution of active contour models, which could avoid large deformation of the adjacent curves and keep the shape conformability of contour in the evolution. The shape prior can be integrated into the block evolution method, which is effective in dealing with missing features of images and noise. The proposed method can be applied to image sequence segmentation. Experiments demonstrate that the proposed method improves the robust performance of active contour models and can increase the flexibility of applications in image sequence segmentation.
Object extraction and image segmentation  are an important and fundamental topics in computer vision and image processing. Snakes or active contour models (ACM)  which have shown their great performances are the key methods for image segmentation. The principal idea in snakes is to obtain an optimum by minimizing an energy functional. A contour is evolved by minimizing some certain energies to match the object boundary while preserving the smoothness of the contour. There are broadly two types of active contour models according to the representation of the curve, that is, parametric active contours  and implicit active contours [3–5]. The active contour is usually represented by landmarks in parametric active contours and an energy functional was originally introduced by Kass et al. , while contours in implicit approaches are represented by level set [3, 5], which offers great flexibility for the curve topology. The numerical computations of evolving level set function can be elegantly performed by using the mature numerical algorithm of partial differential equations (PDE) .
Contours always evolve to major deviations from true object boundary. One of the reasons is that various models could not meet all scenes, such as noise, inhomogeneous intensity [7, 8], and missing features in images; another one of the reasons is the mode of deformation of contours, and contours may suffer from undesired location and could not escape the local minimization [9, 10] because of improper mode of deformation. In implicit active contour model, contour represented by level sets may extract unnecessary objects. Though the energy functionals of some active contour models are convex [11, 12], some unnecessary components (such as noises) are also extracted. For parametric active contour model, contours can be not smooth or appear as self-crossing because of noise or improper parameters. As shown in Figure 1, the image is polluted by noise. The deformation of contour is shown and evolving contour appear as self-crossing within few iterations, which can cause the contour converge to undesired results.
To improve the robustness of active contours, the shape prior [13–17] is often used. The prior knowledge of the shape to be segmented is modeled based on a set of manually annotated shapes to guide the segmentation. In recent works, the shape prior was applied by regularizing the distance from the active contour to the template in a level set framework. The shape prior has proven to be a powerful tool in segmentation. However, the shape prior is learnt from a set of annotated data. As pointed by , existing shapes in the training set are difficult to model the new instance in the testing images. Recently, the active contour with group similarity (ACGS) is proposed by Zhou et al. . In ACGS, the shape prior is not learnt, the shapes of evolving contour in several frame images are utilized to constrain the evolution of contour. ACGS is a better method to evolve the contour, which is effective in dealing with missing features in images.
In this paper, we propose a robust evolution method to keep the similarity among evolving sequence contours for parametric active contours model. The shape consistence of the deforming contour is kept in the evolution. Large deformation of the adjacent curves is avoided. In summary, the contributions of this paper are as follows:
A block evolution mode is proposed, where a block is consisted by contours of former iterations or contours of shape prior.
The proposed algorithm has generality, which could be conveniently integrated into active contour models, including the shape prior based active contour model.
The proposed method also has flexibility, which can be used to image sequence segmentation.
The rest of this paper is organized as follows: Section 2 introduces the model and algorithm of our method. The proposed algorithm is extended to the active contour model with shape prior in Section 3. Section 4 demonstrates the merits of our method by experiments. Finally, Section 5 concludes the paper with some discussions.
2. Proposed Method
In active contour model, an energy functional is usually minimized to segment the objects. For the energy functional with respect to , the curve is viewed as the object when the energy functional obtains its minimum. is minimized as follows:The deformation or evolution equation of curve is obtained with calculus of variations and gradient descent method, that is,where is the gradient of function , is the time variable, and is a parameter. The discrete form of the above equation is as follows:where represents the th iterative solution and is the time step. The converged curve is viewed as the object boundary through a sequence of evolution . From (3), is determined by and the current gradient vector . Because of influences of noises, parameters and nonconvexity of energy functional, the gradient descent direction of may have derivation from ideal gradient direction, which causes to suffer from a local minimum. Various of methods are proposed to define novel energy functional [19–23] or introduce some optimal methods [24–27] for obtaining robust ideal results. However, to our knowledge, few researchers try to change or improve the evolution mode of contours.
2.1. Energy Functional Model
Based on the above analysis, a block evolution method of curve is proposed by imposing a constraint into active contour model. To keep the shape conformability of deforming contours, the following equation is considered:Corresponding to the curve , the set is a block constituted by several curves, and is the cardinality of . is the energy functional of an active contour model to evolve the contour, and . Through a sequence of evolution , the converged contour is viewed as the final result.
The goal is to robustly evolve the curve converging to object. Thus, an energy functional keeping the consistence of evolving contours is integrated into . A nuclear norm , that is, the sum of singular values of is considered. The nuclear norm is a continuous function and convex; some fast algorithms could be utilized. The energy functional of the proposed model is defined as follows:Since is a block including several contours and contour is represented by parametric curve in the plane, , is the number of landmarks, and is a landmark on the curve. Thus, the size of is . From the above equation, because the constraint of is imposed, the shape conformability is kept and the robustness of evolution is improved. is the energy functional of a general active contour model, such as the parametric C-V model . The parametric C-V model is defined as follows:where is the gray intensity, and are the regions inside and outside the contour , and are the mean intensities of and , respectively, represents the length of contour , and is a parameter. is usually less sensitive to initialization and has fewer parameters to tune. makes the contour evolving to object boundary. is a tight convex surrogate to the rank operator, which keeps the elements of similar. With and nuclear norm , contour evolves robustly into the object boundary. Similar to (2), the evolution equation of is computed as follows:where represents the th iterative solution and is the time step. The converged contour is viewed as the final result through a sequence of evolution . From the above equation, the current is determined by and the gradient . Since is block of several contours, the evolution mode is called block evolution method. In a block, the contours of former iterations are utilized to avoid large deformation of evolving contours.
Compared with ACGS, temporal sequence relationship in for dealing with one image with the proposed model existed, while ACGS uses multiple images to evolve contours, respectively, and there is no temporal sequence in evolving contours with ACGS. Thus, the proposed method aims to keep the deformation robustness of contour. Furthermore, the prior information can be flexibly utilized to guide the evolution with the proposed method.
In order to solve (5), a regularized method is considered. is a differentiable function and is a convex function. For the linear combination of and , Proximal Gradient (PG) method is always used to solve the problem. The PG uses quadratic approximation to based on the previous estimate at each iteration.where denotes the Frobenius norm, is a constant, and . By introducing a variable , the above equation becomeswhere is updated by . Therefore, the key problem is to solve (9). For (9), it has been proved that it can be solved by the following expression:where is the singular value thresholding operator; it is given as follows:where and are the left and right singular vectors of , is the singular value, and .
Then the next issue is to find . Since there are temporal sequence relationships in , should be computed orderly. At each iteration, we evolve the active contours according to the image based forces and then impose the block regularization via singular value thresholding. The overall algorithm is summarized in Algorithm 1.
For Algorithm 1, it is not necessary to have explicit representation for the energy function ; the gradient of energy functional is utilized from Algorithm 2. Therefore, block evolution method can be integrated into the active contour model based on external force field [28–32], such as gradient vector field (GVF) , vector field convolution (VFC)  external forces, and just replacing with external force field according to (8). The proposed algorithm has advantage of evolution robustness, since it utilizes the information of previous several iterations.
In the above section, the block evolution method is integrated into active contour model. For the active contour model with shape prior, the block evolution method is also expanded to the active contour model with shape prior. For a shape prior , the energy functional of active contour model with shape prior is usually defined as follows:where is the linear or affine transform, such as translation, scaling, and rotation. It is necessary to compute the parameters of at each iteration. The evolution with shape prior is computed by minimizing the above energy functional:The shape prior gradient is integrated to energy gradient descent direction of to constraint the shape of deforming contour. On the other hand, the above equation can be written as follows:The active contour model with shape prior has better performance because of dynamically utilizing the information of and . Thus, the block evolution method is also integrated into active contour model with shape prior by dynamically using the contour information and . Similar to Algorithm 1, the block evolution algorithm for active contour model with shape prior is given as shown in Algorithm 2.
In fact, it is easy to extend to the active contour model with prior shapes of multiple contours. Corresponding to the program statement of Algorithm 2, the extended algorithm for active contour model with prior shapes is obtained by setting statement of Algorithm 2, where are the prior shapes. In order to keep the consistence, we still use representing and Algorithm 1 is extended to the active contour model with prior shape of multiple contours.
According to the iteration and in proposed algorithm, the is determined by and not only determined by . In the block of , the in block of is the shape prior, which has advantages in dealing with missing features of objects and some complex background of image, such as noise. For sequence image segmentation, the segmented objects for several frames can be viewed as the shape prior. Therefore, the extended Algorithm 2 for active contour model with shape prior can be applied to segmentation for sequence image [18, 33] and object tracking [34–37].
An example is shown to verify the effectiveness of the proposed method, which is shown in Figure 2. Setting the same parameters in Figure 1, active contour model with the block evolution method is used to segment the object. As shown in Figure 2, the parameter is set. The object boundary is extracted with only block iterations. The block evolution method without shape prior is tested, and the nuclear norm is utilized to constraint the shape conformability of evolving contours. The evolution is robust and the object boundary is converged. Compared with the evolution utilizing parametric C-V model, the block evolution method is more robust in the evolution.
4. Experiments and Analysis
To demonstrate the advantages of the block evolution method, we compare the results of the same active contour model before and after applying the proposed constraint. Then, the proposed method is compared with ACGS. If there is no other statement, the region based active contour, that is, the parametric C-V model, is selected as the basic model.
4.1. Experimental Results
In this section, several images are synthesized to test the performances of proposed method. A heart shape is the object. There are occlusions or deletions to be added in these images. The task is to extract the heart shapes in all tested images. Our proposed algorithm and the parametric C-V model are tested. As shown in Figure 3, the result is obtained with the parametric C-V model. The block evolution method with shape prior is shown in Figure 5, and the block evolution method with shape priors is shown in Figure 4. From the comparisons, the block evolution method with multiple shape prior is robust against the occlusion and deletion. In contrast, the parametric C-V model is sensitive to occlusions and missing features in segmenting images. As shown in Figure 3, the shapes of extracted boundaries with the parametric C-V model are different. In the proposed method, the shape constraint is imposed and the shape consistence of evolving contours is kept. This is the reason why the proposed method is robust to occlusions or missing features of images.
The proposed algorithm is especially fit to apply to segmentation of image sequences since the extracted object boundary in the frame is viewed as the prior information in segmenting the next frame. Some image sequences are utilized to test the ability of the proposed method. Two typical image sequences show the performances of the proposed method; there are continuous frames in one image sequence; and there are discontinuous image frames in the other image sequence.
In the first image sequence, the continuous ten image frames are selected, which are shown in Figure 6, and the size of each image in this sequence is . The results with the proposed method and the parametric C-V model are shown in Figures 8 and 7, respectively. The extracted object boundaries with both methods are compared in Figure 9. The fish in these images is blurry, which is difficult to extract. From the results in Figures 8 and 7, respectively, both of these tested methods roughly converge to the contour of the object. However, there are still some differences between the extracted results. As shown in Figure 9, (a) are the results with the parametric C-V model and (b) are the results with the proposed method. The converged contours are nonsmooth, appear as double-contours and self-crossing, which is clearly shown in Figure 10(a). While the proposed method utilizes the shape information to constrain the evolution of contours, the extracted contour is viewed as the shape prior in the next image frame. Because of the constraint of shapes, the shapes of evolving contour are always smooth in the evolution, which is shown in the second row of Figure 10.
Because of the nuclear norm to constrain the evolving contours, the shapes of the deforming contour keep consistence throughout evolution of contours. Therefore, the converged contour is always smooth, as shown in the second row of Figure 10. On the other hand, since utilizing the shape information and nuclear norm, the proposed algorithm is robust and the accuracy of extracted results with proposed method is more precious compared with the typical active contour model. According to Figure 10, the enlarged results are shown; it seems that the proposed method obtains better object boundaries compared with the tested method. In order to evaluate the performances and accuracies of tested methods effectively, the quantitative results are given. For each tested algorithm, the traditional -measure score  is the weighted, harmonic mean of precision and recall values; it is usually used to evaluate the quantitative results. It is given as follows:The average -measure score (mean standard deviation) for each algorithm is the following: the tested method: 0.82 ± 0.08 and proposed method: . According to score, the proposed method outperforms the parametric C-V model in extracting object boundary.
4.2. Comparison with ACGS in Sequence Image Segmentation
In this part, the proposed algorithm and ACGS are tested and compared. As analyzed in the Section 3, because of integrating prior information, the proposed method has better flexibility in segmentation of image sequence. Eight images are continuously selected to form an image sequence to test algorithms, as shown in Figure 11. The object in these images is a ball and the locations of ball in each image frame are different; the shapes of objects in these images are the same. ACGS offers an effective way to image sequence segmentation. In ACGS, the contours must be simultaneously initialized in every tested sequence images. Thus, the location information of objects in the current images is useless in the following images. Once the initialization of contours is undesired, the results with ACGS is always unsatisfactory. As shown in the Figure 12(a), ACGS failed to extract the objects in some images. A typical example is the first image of second row in Figure 12(a), and the contour is completely outside the object. As shown in Figure 12(b), ACGS with careful initialization obtains better results compared with Figure 12(a). By contrast, the proposed method obtains better results, which is shown in Figure 12(c). The proposed method allows the final result in current frame acting initial contour in the next frame. Therefore, the proposed method always obtains better accuracy of extracting objects compared with ACGS.
In this paper, a method of active contour with block evolution method is proposed. In the proposed method, the shape prior can be integrated, and the shapes similarity of evolving contours are kept. On the other hand, evolving contour deforms robustly with the proposed method because of utilizing the contour’s evolution result of the previous iteration steps. Simulations show that the proposed method has the robust performances in segmenting objects. Compared with the state-of-the-art method (ACGS), the proposed method has shown flexibility of application in segmentation of image sequence.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
This work is supported by the National Natural Science Foundation of China (no. U1404603).
S. C. Zhu and A. Yuille, “Region competition: unifying Snakes, Region Growing, and Bayes/MDL for multiband image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 9, pp. 884–900, 1996.View at: Publisher Site | Google Scholar
M. Kass, A. Witkin, and D. Terzopoulus, “Snakes: active contour model,” International Journal of Computer Vision, vol. 1, no. 4, pp. 321–331, 1988.View at: Google Scholar
S. Osher and J. A. Sethian, “Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations,” Journal of Computational Physics, vol. 79, no. 1, pp. 12–49, 1988.View at: Publisher Site | Google Scholar | MathSciNet
R. Kimmel, A. Amir, and A. M. Bruckstein, “Finding shortest paths on surfaces using level sets propagation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 6, pp. 635–640, 1995.View at: Publisher Site | Google Scholar
L. A. Vese and T. F. Chan, “A multiphase level set framework for image segmentation using the Mumford and Shah model,” International Journal of Computer Vision, vol. 50, no. 3, pp. 271–293, 2002.View at: Publisher Site | Google Scholar
G. Aubert and P. Kornprobst, Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations, Springer, NY, USA, 2006.View at: MathSciNet
C. Li, C.-Y. Kao, J. C. Gore, and Z. Ding, “Implicit active contours driven by local binary fitting energy,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '07), Minneapolis, USA, June 2007.View at: Publisher Site | Google Scholar
F. Dong, Z. Chen, and J. Wang, “A new level set method for inhomogeneous image segmentation,” Image and Vision Computing, vol. 31, no. 10, pp. 809–822, 2013.View at: Publisher Site | Google Scholar
C. Han, T. S. Hatsukami, J.-N. Hwang, and C. Yuan, “A fast minimal path active contour model,” IEEE Transactions on Image Processing, vol. 10, no. 6, pp. 865–873, 2001.View at: Publisher Site | Google Scholar
A. K. Mishra, P. W. Fieguth, and D. A. Clausi, “Decoupled active contour (DAC) for boundary detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 2, pp. 310–324, 2011.View at: Publisher Site | Google Scholar
X. Bresson, S. Esedoglu, P. Vandergheynst, J.-P. Thiran, and S. Osher, “Fast global minimization of the active contour/snake model,” Journal of Mathematical Imaging and Vision, vol. 28, no. 2, pp. 151–167, 2007.View at: Publisher Site | Google Scholar | MathSciNet
E. S. Brown, T. F. Chan, and X. Bresson, “Completely convex formulation of the Chan-Vese image segmentation model,” International Journal of Computer Vision, vol. 98, no. 1, pp. 103–121, 2012.View at: Publisher Site | Google Scholar | MathSciNet
N. Paragios, “A level set approach for shape-driven segmentation and tracking of the left ventricle,” IEEE Transactions on Medical Imaging, vol. 22, no. 6, pp. 773–776, 2003.View at: Publisher Site | Google Scholar
S. Zhang, Y. Zhan, M. Dewan, J. Huang, D. N. Metaxas, and X. S. Zhou, “Towards robust and effective shape modeling: sparse shape composition,” Medical Image Analysis, vol. 16, no. 1, pp. 265–277, 2012.View at: Publisher Site | Google Scholar
D. Cremers, “Dynamical statistical shape priors for level set-based tracking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 8, pp. 1262–1273, 2006.View at: Publisher Site | Google Scholar
F. Chen, H. Yu, R. Hu, and X. Zeng, “Deep learning shape priors for object segmentation,” in Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition, (CVPR '13), pp. 1870–1877, USA, June 2013.View at: Publisher Site | Google Scholar
T.-T. Tran, V.-T. Pham, and K.-K. Shyu, “Moment-based alignment for shape prior with variational B-spline level set,” Machine Vision and Applications, vol. 24, no. 5, pp. 1075–1091, 2013.View at: Publisher Site | Google Scholar
X. Zhou, X. Huang, J. S. Duncan, and W. Yu, “Active contours with group similarity,” in Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition, (CVPR '13), pp. 2969–2976, USA, June 2013.View at: Publisher Site | Google Scholar
V. Caselles, R. Kimmel, and G. Sapiro, “Geodesic active contours,” International Journal of Computer Vision, vol. 22, no. 1, pp. 61–79, 1997.View at: Publisher Site | Google Scholar
C. Li, C. Xu, C. Gui, and M. D. Fox, “Level set evolution without re-initialization: a new variational formulation,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '05), pp. 430–436, June 2005.View at: Publisher Site | Google Scholar
B. Wang, X. Gao, D. Tao, and X. Li, “A nonlinear adaptive level set for image segmentation,” IEEE Transactions on Cybernetics, vol. 44, no. 3, pp. 418–428, 2014.View at: Publisher Site | Google Scholar
K. Zhang, L. Zhang, K.-M. Lam, and D. Zhang, “A level set approach to image segmentation with intensity inhomogeneity,” IEEE Transactions on Cybernetics, vol. 46, no. 2, pp. 546–557, 2016.View at: Publisher Site | Google Scholar
T. F. Chan and L. A. Vese, “Active contours without edges,” IEEE Transactions on Image Processing, vol. 10, no. 2, pp. 266–277, 2001.View at: Publisher Site | Google Scholar
A. A. Amini, T. E. Weymouth, and R. C. Jain, “Using dynamic programming for solving variational problems in vision,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 9, pp. 855–867, 1990.View at: Publisher Site | Google Scholar
S. Zhang, Y. Zhan, and D. N. Metaxas, “Deformable segmentation via sparse representation and dictionary learning,” Medical Image Analysis, vol. 16, no. 7, pp. 1385–1396, 2012.View at: Publisher Site | Google Scholar
J. Peng, J. Shen, A. Yao, and X. Li, “Superpixel optimization using higher order energy,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 5, pp. 917–927, 2016.View at: Publisher Site | Google Scholar
J. Shen, Y. Du, and X. Li, “Interactive segmentation using constrained laplacian optimization,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 24, no. 7, pp. 1088–1100, 2014.View at: Publisher Site | Google Scholar
C. Xu and J. L. Prince, “Snakes, shapes, and gradient vector flow,” IEEE Transactions on Image Processing, vol. 7, no. 3, pp. 359–369, 1998.View at: Publisher Site | Google Scholar | MathSciNet
X. Xie and M. Mirmehdi, “MAC: magnetostatic active contour model,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 4, pp. 632–646, 2008.View at: Publisher Site | Google Scholar
B. Li and S. T. Acton, “Active contour external force using vector field convolution for image segmentation,” IEEE Transactions on Image Processing, vol. 16, no. 8, pp. 2096–2106, 2007.View at: Publisher Site | Google Scholar | MathSciNet
C. Xu and J. Prince, “Gradient Vector Flow: A new external force for snakes,” in Proceeding of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 66–71, San Juan, Puerto Rico, USA, 1997.View at: Google Scholar
C. Xu and J. L. Prince, “Generalized gradient vector flow external forces for active contours,” Signal Processing, vol. 71, no. 2, pp. 131–139, 1998.View at: Publisher Site | Google Scholar
W. Tao, K. Li, and K. Sun, “SaCoseg: object cosegmentation by shape conformability,” IEEE Transactions on Image Processing, vol. 24, no. 3, pp. 943–955, 2015.View at: Publisher Site | Google Scholar | MathSciNet
Y. Wu, B. Ma, M. Yang, J. Zhang, and Y. Jia, “Metric learning based structural appearance model for robust visual tracking,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 24, no. 5, pp. 865–877, 2014.View at: Publisher Site | Google Scholar
J. Ning, L. Zhang, D. Zhang, and W. Yu, “Joint registration and active contour segmentation for object tracking,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 9, pp. 1589–1597, 2013.View at: Publisher Site | Google Scholar
K. Zhang, L. Zhang, M.-H. Yang, and Q. Hu, “Robust object tracking via active feature selection,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 11, pp. 1957–1967, 2013.View at: Publisher Site | Google Scholar
L. Qin, C. Zhu, Y. Zhao, H. Bai, and H. Tian, “Generalized gradient vector flow for snakes: New observations, analysis, and improvement,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 5, pp. 883–897, 2013.View at: Publisher Site | Google Scholar
A. Kovacs and T. Sziranyi, “Harris function based active contour external force for image segmentation,” Pattern Recognition Letters, vol. 33, no. 9, pp. 1180–1187, 2012.View at: Publisher Site | Google Scholar