Abstract
Image segmentation is an essential task in computer vision and pattern recognition. There are two key challenges for image segmentation. One is to find the most discriminative image feature set to get high-quality segments. The other is to achieve good performance among various images. In this paper, we firstly propose a selective feature fusion algorithm to choose the best feature set by evaluating the results of presegmentation. Specifically, the proposed method fuses selected features and applies the fused features to region growing segmentation algorithm. To get better segments on different images, we further develop an algorithm to change threshold adaptively for each image by measuring the size of the region. The adaptive threshold can achieve better performance on each image than fixed threshold. Experimental results demonstrate that our method improves the performance of traditional region growing by selective feature fusion and adaptive threshold. Moreover, our proposed algorithm obtains promising results and outperforms some popular approaches.
1. Introduction
Image segmentation plays a significant role in computer vision and can be applied to various fields like region proposal generation [1–3], face recognition [4], and disease detection [5–8]. There are many kinds of image segmentation algorithms, such as edge-based, region-based, threshold-based, and graph-based image segmentation algorithm. Algorithms based on edges [7, 9, 10] achieve good performance on images where the boundary of the object is distinct, but these methods are less resistant to noise and require higher image quality. Region-based [11–13] and threshold-based [14–16] segmentation methods merge pixels into regions by their features like color, texture, or their combination. However, for a certain image, there are some redundant features which do not play a role in segmentation. Merging these features with others demands a large amount of computation. Algorithms based on graph theory [8, 17, 18] often need foreground or background seeds to initialize the energy function. The seeds are often labeled manually, which means graph-based methods may not segment images automatically. Thus, a method which can choose not only the best feature combination but also the optimal threshold is in demand.
Recent years have witnessed the emergence of various feature selection and self-adaptive image segmentation algorithms. Feature selection algorithms [19, 20] can select an effective feature subset rather than the whole feature set so as to reduce time and space consumption. Most adaptive algorithms focus on detecting cluster centers of pixels adaptively [21], changing pixels’ local information during segmenting [22], or determining the best threshold [14, 15]. However, few works focus on how to combine features and choose threshold adaptively according to different images, which may achieve accurate segmentation performance for images with high variations. Our intuition is that, for a certain image, merging some kinds of features will help to produce higher-quality image segments than using only one kind of feature, yet merging other features will not. To fully utilize multiple features of a certain image, it is crucial to select and combine the most discriminative features. After obtaining the best feature combination, the most important challenge for region-based segmentation algorithm is to determine the optimal threshold that decides whether to merge small regions or not.
We have previously introduced an optimized automatic seeded region growing (OASRG) [23] algorithm that locates seeds by affinity propagation (AP) clustering algorithm and grows regions by color and edge features. Although the algorithm can segment images without human interaction, it cannot adjust threshold adaptively or select best feature combination according to different images. Based on OASRG algorithm, we further propose a selective feature fusion based image segmentation (SFF-IS) algorithm to improve the performance of previous works. In order to segment images automatically, SFF-IS algorithm inherits the method of finding seeds and extracting features from OASRG algorithm. Distinct from preceding reports, SFF-IS algorithm involves (1) adding textural features to original features, (2) selecting effective feature combination adaptively for better performance, and (3) changing the threshold during region growing according to different images.
The performance of our algorithm is quantitatively measured on PASCAL VOC 2007 segmentation dataset [24]. We use LAB, LBP, Canny features, and their combination to help to segment images. Compared to OASRG algorithm, our method improves the performance by the selective feature fusion and the adaptive threshold. Moreover, experimental results demonstrate that our algorithm can produce superior image segments than some popular approaches.
The rest of this paper is organized as follows. In the part of Related Work, we will introduce previous works on feature selection and adaptive segmentation. In the part of Our Method, we will describe the pipeline of our algorithm in detail. In the part of Experiments, we will introduce experiments, including experimental settings and results. Finally, we will conclude this paper in the part of Conclusion
2. Related Work
2.1. Image Segmentation
In the past, some works have emerged as the state-of-the-art image segmentation methods. Watershed [25] and its improved version [26] performed a gradient ascent starting from local minima to produce segments, which were often highly irregular in shape and size. Deng et al. [27] proposed the joint systems engineering group (JSEG) algorithm, achieving consistent segmentation and tracking results on real images and videos. Comaniciu et al. [28] proposed a general nonparametric technique called Mean Shift. Felzenszwalb et al. [29] introduced a graph-based image segmentation algorithm which reserved details in low-variability regions while ignoring details in high-variability regions. Most recently, a certain degree of attention was given to hybrid features that combined two or more features. Arbelez et al. [10] developed the spectral clustering-based contour detector, and the output of the contour detector was combined with brightness, color, and texture gradients to generate a hierarchical region tree. Achanta et al. [30, 31] introduced a simple linear iterative clustering (SLIC) algorithm to improve segmentation performance with a good balance between speed and accuracy. Storath et al. [32] proposed a fast splitting approach to solve image segmentation problem, producing results of a quality comparable with that of graph cuts and the convex relaxation strategies.
2.2. Feature Selection in Image Segmentation
Feature selection was an important task in machine learning to increasing the performance of an algorithm [33]. The huge amounts of features described images in detail. On the other hand, not all features were essential for segmentation, since many of them were redundant or even irrelevant [34]. These redundant features might degrade the performance. Luo et al. [35] developed an effective feature selection method which selected a group of mixed color features or channels according to the principle of the least entropy of pixels frequency histogram distribution. Liang et al. [18] developed a genetic programming based segmentation method, achieving good performance on distinguishing object or background. Feng et al. [17] proposed the interactive segmentation algorithm which selected a single feature to determine a pixel’s label locally. Their algorithm performed better on RGBD images with fewer user inputs. Cheng et al. [19] developed the hierarchical feature selection method which demonstrated its advantage in speed and segmentation quality. In this paper, we propose a selective feature fusion algorithm to choose the best feature set by evaluating the results of presegmentation. Our proposed algorithm fuses selected features and applies the fused features to image segmentation algorithm.
2.3. Adaptive Segmentation
The methods which adjusted parameters to segment different images were collectively called adaptive segmentation. The adaptive parameters were always used in clustering-based image segmentation to find the number of clusters and the correspond centers. In this spirit, some works aimed to develop adaptive clustering-based segmentation methods, such as [21, 22]. For threshold-based methods, there were always different thresholds for different kinds of data. Hence, automatically determining the optimal threshold was especially significant for segmentation. The adaptive threshold-based algorithm in [14] adjusted threshold while segmenting an image, which took the best of local and global image information. Guo et al. [15] presented a self-adaptive threshold approach. The threshold was obtained with two-dimensional entropy and optimized with different evolution. To get the best segments on each image, we propose an adaptive segmentation algorithm which changes the threshold during region growing according to different images.
3. Our Method
The SFF-IS algorithm proposed in this paper divides an image into multiple regions by aggregating pixels with similar characteristics around seed pixels. The pipeline of this method includes five major steps, which are shown in Figure 1. Firstly, for a given image, the seed pixels are automatically located by affinity propagation clustering algorithm. Secondly, features including color, texture, and edge are extracted for each pixel. Thirdly, a feature selector based on feedback judges which features will help to achieve better segmentation performance. Fourthly, an optimal threshold is calculated adaptively according to selected features. Finally, based on the feature combination of each pixel and the optimal threshold, a mask of segmented image is generated by region growing.
For region growing algorithm, the location of seeds is of significance. A seed point usually shares high similarity with its surrounding pixels and can represent the region where it grows. This characteristic of seed point is very similar to the cluster center that represents its cluster. In this spirit, we propose the automatic seeded (AS) algorithm to locate seed points in [23]. The AS method firstly divides an image into super-pixel blocks and then utilizes affinity propagation clustering algorithm [36] on these blocks. The experimental result in [23] shows that our automatic seeded algorithm is more efficient than the manual seeded algorithm.
3.1. Feature Extraction
It is known to all that different features describe different characteristics of an image. Thus, it makes sense to extract various features to better depict the image. Figure 2 shows several mask results generated by segmenting different images with their different features, including color, texture, and edges feature.
Color is one of the most widely used features in image segmentation. Of all color spaces, RGB is one of the most commonly used color models. But for image segmentation task, CIELAB color space has been proved to be very useful due to its consistency with the human visual system [10].
The texture is another helpful property for segmentation, especially when they are combined with other features. Local binary patterns (LBP) [37] is a kind of powerful texture feature which emphasizes the local structure and thus has the advantage of robustness to rotation and nonuniform illumination [38].
Edge detectors produce contours of object. Canny edge detector augments itself with nonmaximum suppression and adaptive threshold with hysteresis [39]. Although these contours can not guarantee to be closed, they can provide a perfect complement to other features.
Therefore, LAB color, LBP texture, and Canny edge features will be extracted and fused for image segmentation.
3.2. Selective Feature Fusion
It is well known that different feature combination can provide different information. However, many features are redundant or even irrelevant for image segmentation. Selective feature fusion focuses on selecting the most relevant feature subset for different images and merging these features.
Our intuition is that the best feature set varies from different images. We find that color feature does well on most images, but is useless for some images whose colors are discontinuous and various. Texture feature, as a supplement, is helpful to segment these pictures. For Canny feature, the formula is given to calculate the importance of Canny edge (see (1)).where and represent the pixel and . is a set of pixels belonging to Canny edge. is a set of pixels belonging to an image.
Based on above idea, we propose a selective feature fusion (SFF) algorithm (see Algorithm 1) to choose which features are used to segment a certain image.
| ||||||||||||||||||||||||||||
3.3. Adaptive Threshold
Region growing calculates feature similarity between seed point and its neighbor point. By comparing similarity and threshold, the algorithm decides neighbor point whether to be merged or not. Therefore, the threshold is an important parameter in region growing. The fixed threshold lacks generalization ability and cannot achieve accurate segmentation for images with high variations. Therefore, we present an adaptive threshold method (see Algorithm 2). The similarity between two pixels is calculated by function (see (2))where and are the -th dimension of features and . is the dimension of the feature space. is a similarity function between features and . The adaptive threshold is calculated by where and are features of and . represents the region grown from seed . is the number of pixels in region .
3.4. Selective Feature Fusion Based Image Segmentation
To automatically find the effective feature subset for different images, we propose a selective feature fusion based image segmentation (SFF-IS) algorithm in this section. The SFF-IS algorithm uses feature subset rather than all features to segment images and constantly merge seed pixels with their surrounding pixels by their similarity. The decision is made by calculating the value of by where and represent seed point and its neighbor point. , , and are selection results, where means that corresponding feature is selected. and are features of and . and are thresholds. is a set of pixels which belong to Canny edge. represents merging pixel and pixel . If a point belongs to , the point is an edge point and can not be merged. The segmentation process is described as shown in Algorithm 3.
| ||||||||||||||||||||||||||||||
4. Experiments
4.1. Feature Selection
We evaluate our algorithms on the PASCAL VOC 2007 segmentation dataset [24]. We use LAB, LBP, Canny features, and their combination to segment images and calculate Absolute Adjusted Rand Index (AARI) [40] to measure the pairwise similarity between proposed segmentation and ground truth. The AARI ranges from to .
The graphical results of five example images are shown in Figure 3. It can be seen that the aeroplane in No. 005043 image is not segmented by LAB or Canny, neither is the No. 005043. By combining two or three features, the object is easier to be distinguished from its background. Moreover, SFF algorithm applies different feature combinations on the basis of different images, which helps to generate higher-quality segmentation masks than those of Merge feature.
(a) Original images
(b) Segmentation with LAB features
(c) Segmentation with LBP features
(d) Segmentation with Can features
(e) Segmentation with LL features
(f) Segmentation with LA+C features
(g) Segmentation with LB+C features
(h) Segmentation with Merge features
(i) Segmentation with SSF features
Figure 4 shows the AARI values of five example pictures segmented by different feature combinations. We can see from Figure 4 that a single feature can not segment image sometimes. For example, LAB color feature does not segment No. 005043 at all, getting the lowest AARI in result. Canny feature has the same performance in No. 005043 and No. 006946. Fusing two features can increase the value of AARI, but different combinations have different influence on the same image. For No. 002619, LL and LA+C both improve the performance dramatically, while LB+C only gets the same value of AARI as LBP. On the other hand, Merge feature which consists of all three features does not get the highest AARI among five images, while SFF feature does well on all five images by choosing the effective feature subset. In conclusion, merging more features do not promise to get better performance, and selecting the best feature combinations will produce accurate segments.
To confirm our hypothesis, we conduct the same experiment on training data with and without SFF algorithm and then calculate the average AARI for each method. The result is shown in Figure 5. LAB, LBP, and Canny features get very small AARI value, because each of them cannot depict an image in detail. In addition, merging three features does not get better performance than two features, which proves the conclusion mentioned earlier. On the whole dataset, LA+C produces comparable segments to SFF method. One way or another, the average AARI of segmentation with SFF algorithm is the highest, outperforming the second best by . The results show that SFF does improve the segmentation accuracy.
4.2. Adaptive Threshold
As we describe above, the adaptive threshold has to be initialized. In [23], our experiment showed that was the best threshold for LAB color feature. In this section, we use the same way to explore the optimal thresholds and for LBP and LAB+LBP. We segment training data with different thresholds and calculate average AARI accordingly. Figure 6 shows that the average AARI varies with thresholds. In addition, and are the optimal thresholds separately for LBP and LAB+LBP. Therefore, we set , , and to initialize the thresholds.
Then, we apply the adaptive threshold algorithm to different combinations of features and calculate the AARI for segmentation results. Figure 7 shows that our adaptive threshold algorithm achieves better performance than the fixed threshold on four kinds of feature combinations. The experimental results demonstrate that our method can improve their performance when plugged into other threshold-based approaches.
4.3. Image Segmentation
We compare the performance of our algorithm with other methods: OASRG [23], WS [25], GBIS [29], JSEG [27], SLIC [30], GPU-SLIC [31], and FPVVI [32]. We use the AARI [40] and the Segmentation Covering (Covering) [10] to measure the segmentation results. The Covering aims at evaluating the average overlap between proposed segmentation and ground truth. Figure 8 shows that our algorithm outperforms other approaches on both the AARI and the Covering. Equipped by selective feature selection method and adaptive threshold, the performance of original method has been improved. In conclusion, our proposed algorithm is testified that it can produce high-quality image segments in most case.
5. Conclusion
In this paper, we propose a selective feature fusion based image segmentation (SFF-IS) algorithm with an adaptive threshold, which consists of seed determination, feature extraction, selective feature fusion, adaptive threshold determination, and region growing based segmentation. Our algorithm can automatically select an effective feature subset for different images and adaptively change the threshold during region growing. We conduct experiments on PASCAL VOC 2007 dataset. The results demonstrate that our algorithm proposed in this paper improves the performance of previous works and outperforms other segmentation approaches.
The contribution of our proposed algorithm involves (1) fully utilizing image features by selecting an effective feature subset for better performance and (2) adaptively changing the threshold during region growing according to different feature combination, thus achieving better segmentation performance for different images.
Feature extraction is an important procedure for image segmentation. In future work, we will extract more features to provide more information for image segmentation and extend our selective feature fusion algorithm on those features. As feature extraction using deep learning has achieved good results for image segmentation, our future work will focus on integrating our method with deep learning model.
Data Availability
Previously reported PASCAL VOC 2007 data were used to support this study and are available at http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html or http://host.robots.ox.ac.uk/pascal/VOC/voc2007/index.html. These prior studies (and datasets) are cited at relevant places within the text as [24].
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work is partially supported by the National Key Research and Development Project under Grant no. 213, the National Natural Science Foundation of China under Grant no. 61573259, and the Special Project of Ministry of Public Security under Grant no. 20170004.