Journal of Electrical and Computer Engineering

Journal of Electrical and Computer Engineering / 2017 / Article

Research Article | Open Access

Volume 2017 |Article ID 7347421 |

Huidong He, Xiaoqian Mao, Wei Li, Linwei Niu, Genshe Chen, "Visual Sensor Based Image Segmentation by Fuzzy Classification and Subregion Merge", Journal of Electrical and Computer Engineering, vol. 2017, Article ID 7347421, 15 pages, 2017.

Visual Sensor Based Image Segmentation by Fuzzy Classification and Subregion Merge

Academic Editor: Ping Feng Pai
Received16 Jun 2017
Revised04 Aug 2017
Accepted09 Aug 2017
Published25 Sep 2017


The extraction and tracking of targets in an image shot by visual sensors have been studied extensively. The technology of image segmentation plays an important role in such tracking systems. This paper presents a new approach to color image segmentation based on fuzzy color extractor (FCE). Different from many existing methods, the proposed approach provides a new classification of pixels in a source color image which usually classifies an individual pixel into several subimages by fuzzy sets. This approach shows two unique features: the spatial proximity and color similarity, and it mainly consists of two algorithms: CreateSubImage and MergeSubImage. We apply the FCE to segment colors of the test images from the database at UC Berkeley in the RGB, HSV, and YUV, the three different color spaces. The comparative studies show that the FCE applied in the RGB space is superior to the HSV and YUV spaces. Finally, we compare the segmentation effect with Canny edge detection and Log edge detection algorithms. The results show that the FCE-based approach performs best in the color image segmentation.

1. Introduction

Image segmentation is a fundamental issue in image processing and computer vision as it is a preliminary process in many applications that need to segment an image into meaningful regions. Applications of segmenting gray-level images are very limited so studies on color image segmentation become interesting because color images provide much more information than gray-level ones. A number of image segmentation techniques have been proposed in past decades [13]. A physical model in [4] is proposed for color image understanding. A widely used technique for color image segmentation is based on the idea of region growing. A color segmentation algorithm combining region growing and region merging processes is discussed in [5]. The fuzzy growing and expanding approach presented in [6] uses histogram tables for fine segmentation. A method for segmentation of color texture region using the defined class maps is presented in [7]. An approach to contrast analysis using fuzzy region growing is documented in [8]. Color image segmentation using seeded region growing is presented in [911]. Most pixel clustering algorithms work in the RGB color space in which the pixels are represented by red, green, and blue components. Clustering methods [12, 13], for example, -means fuzzy classification [14], are often considered as an unsupervised classification of pixels as a prior knowledge of the image. Both supervised and unsupervised segmentation of textured color images are presented in [15, 16]. The level setting approaches to process a color image for segmentation are discussed in [17, 18]. All of the proposed techniques above provide a crisp segmentation of images, where each pixel is classified into a unique subset. This classification may not reflect an understanding of images by humans very well, as the work [1] stated “the image segmentation problem is basically one of psychophysical perception, and therefore not susceptible to a purely analytical solution.” Probably, one of the difficulties in color image segmentation is how to define color conception, since an understanding of colors in an image varies by different observers.

Considering uncertainty and ambiguity in many real applications, this paper presents an approach to color image segmentation based on a fuzzy color extractor (FCE). Its preliminary version appears in [19]. The FCE is directly extended from the fuzzy gray-level extractor, which was applied to recognize landmarks on roads for autonomous vehicle navigation in [2022]; that is, the fuzzy rules for extracting a road from a gray-level image are as follows:If the gray level of a pixel is close to the gray-level of the road,Then one can make it black, orElse one can make it to white.If these are modified to the rules for extracting an object from a color image, thenif the color components of a pixel closely matches the color components of an object,then one can extract this pixel as a Matched pixel of the object, orelse one can extract this pixel as an Unmatched pixel.

The FCE viewed as a mapping operator of measuring colors is able to extract colors that are similar to a color pattern (seed). If a pixel is extracted it is defined as a Matched pixel; otherwise it is defined as an Unmatched pixel. In doing this, we develop procedures IniSeedList and UpdateSeedList for automatic selection of seeds. Our approach mainly consists of two algorithms: CreateSubImage and MergeSubImage. The CreateSubImage algorithm always extracts the seed as the first Matched pixel into a subimage and continually uses the FCE to check neighbors of each Matched pixel to grow the subimage. The subimage is created when neighbors’ colors of all Matched pixels are “not close to” the seed’s color. The CreateSubImage algorithm iteratively invokes UpdateSeedList to select new seeds to create next subimages until all the pixels in the image are processed. Different from all of the color segmentation techniques discussed above, the CreateSubImage algorithm most likely classifies an individual pixel into several subimages by fuzzy sets. As a result, two created subimages may share a common area, which can be considered as a color similarity metric for merging the subimages. The MergeSubImage algorithm first removes insignificant subimages from the set of subimages and merges significant subimages in order to extract meaningful segmented images. The insignificant subimages usually have a “too small” or a “too large” ratio of their area sizes over the source image. A “too small” insignificant subimage is mainly caused by a tiny fragment or a noise pixel in the source image, while a “too large” insignificant image is created by a “bad” seed. Then, the algorithm merges two subimages together according to their color similarity described by the size of their common pixels. We tested the proposed approach using exemplar color images from the database at UC Berkeley [23] in three different color spaces and discussed its effectiveness and robustness by comparing segmentation effect with two edge detection algorithms, namely, Canny edge detection and Log edge detection.

2. Three Color Spaces

In many color-based computer vision applications, it is very important to segment colors in an image, but it is very difficult due to the fact that colors are ambiguous in uncertain environments. In this study, we apply the FCE to segment colors in the image represented in the RGB, HSV, and YUV spaces.

2.1. RGB Color Space

RGB is the most traditional color space and is widely used in image segmentation because of its simplicity and fast processing [24, 25]. Colors consist of red (R), green (G), and blue (B) components, as shown in Figure 1(a).

2.2. HSV Color Space

HSV is another color space widely used for image processing, where a color is decomposed into hue (H), saturation (S), and value (V). The distribution of color in the HSV color space is shown in Figure 1(b). The color components in the RGB space can be transformed into the HSV space by

Usually, the RGB values are between 0 and 255, so the HSV values are also between 0 and 255 through the transformation by (1).

2.3. YUV Color Space

YUV is the color space which is used in an analog color TV system, which translates color image components obtained in the RGB space into the luminance Y and the colors B -Y (U) and R - Y (V) by

The YUV values are between 0 and 255 through the transformation by (2). The advantage of using the YUV color space for color segmentation is that the luminance Y is separated from the colors U and V.

3. Fuzzy Color Extractor

The fuzzy color extractor proposed here enables us to deal with such uncertainty. We will only discuss how to use the FCE-based algorithm defined in the RGB color space to segment color images, because the algorithm can directly process the color images in the HSV and YUV spaces, which are converted from the RGB space. Figure 1(a) shows the RGB space, where colors are represented by their red, green, and blue components in an orthogonal Cartesian space. The upper-left corner in the image is the origin. The color of each pixel denoted by is processed to separate its red, green, and blue components (, , ). The FCE extracts a cluster of colors based on a defined color pattern (seed) denoted by , and is either directly defined by its RGB components (, , ) or determined by a pixel in the image. The color components’ differences between a pixel and are calculated as follows: and represent the size of an array which holds an image. The corresponding fuzzy rules can be obtained as follows:If and and are ,Then is .If or or is or ,Then is .

Both the rules indicate that belongs to the object to be extracted, if the Euclidean distances between and along the three axes in RGB coordinate system are small enough; otherwise, does not belong to the object.

Figure 2(a) shows the membership functions (, , ) for the input fuzzy variables (Negative, Zero, Positive) defined by and Figure 2(b) shows the membership functions (, ) for the output fuzzy variables (Matched, Unmatched) defined by where . Based on , , and , the fuzzy rules produce the weight for Matched and the weight for Unmatched by Figure 2(b) shows the produced areas in the output domain, while and cut and . A crisp output value, , is calculated by the centroid defuzzification method:where represents the envelope function of the areas cut by and in fuzzy output domain. If ,   is extracted, is not extracted, where is a threshold. The FCE can be understood as a mapping operator between Euclidean distances , , in the RGB space and a difference in the intensity space under a fuzzy metric.

The example in Figure 3 shows how to create some subimages with interesting colors from the source image by the FCE. First, we define three seeds: = (255, 0, 0), = (0, 255, 0), and = (0, 0, 255) to create three subimages , , that hold the words “red,” “green,” and “blue” from the source image , as shown in Figure 3(b). Also, we select a seed = (6, 6, 8) at the origin to remove the background from the source image as shown in Figure 3(c). Figure 3(d) shows that holds the color components of the flower petal.

4. Creating Subimages Using FCE

This section discusses the CreateSubImage algorithm for creating a set of subimages from a color image using the FCE in the RGB space. A critical step in this algorithm is to select seeds for the FCE. For this purpose, we evenly divide the RGB space into the eight subspaces _black, _red, _green, _blue, _yellow, _pink, _cyan, _white, which are defined by the well-defined colors: black, red, green, blue, yellow, pink, cyan, white, at their corresponding vertices of the RGB space, as shown in Figure 1(a). Table 1 lists their definitions; for example, _black is defined by a cubic on the red-axis, green-axis, and blue-axis.

Vertex SubspaceRed-axisGreen-axisBlue-axis

(0, 0, 0)_black
(0, 0, 255)_blue
(0, 255, 0)_green
(0, 255, 255)_cyan
(255, 0, 0)_red
(255, 0, 255)_pink
(255, 255, 0)_yellow
(255, 255, 255)_white

We calculate the distances of each pixel’s RGB components to the eight well-defined colors bywhere and are the index of a pixel in the image, is the color components of the pixel located at , is the color components of vertex in the RGB space, and is the distance between the pixel’s color and vertex .

We developed two procedures, IniSeedList and UpdateSeedList, to initialize a seed list and to update it. For an input image, IniSeedList classifies each pixel into one of the eight subspaces according to its minimum distance to vertex that is a well-defined color in the RGB space. In each subspace, IniSeedList selects such a pixel as a seed candidate, seed_can, that has the shortest distance to its vertex . Then, IniSeedList inserts seed_can into the seed list in order of the number of pixels classified in the subspaces and returns it. UpdateSeedList checks if seed_can is popped out or is extracted by the FCE. If so, UpdateSeedList updates seed_can in its corresponding subspace. The pseudocode of the IniSeedList procedures is shown in Pseudocode 1 and the pseudocode of the UpdateSeedList is shown in Pseudocode 2.

IniSeedList ()
for 1 to 8 do
pixel_nu 0;
for 1 to do
for 1 to do
Calculate in (8);
pixel_nu pixel_nu;
Find seed_can in each subspace;
// with the shortest distance to its vertex .
Push seed_can into SeedList;
Sort seed_can;
// according to pixel number classified in the subspaces.
return SeedList;
UpdateSeedList (, SeedList)
subspace_index () 0;
for 1 to 8 do
if (seed_can )
subspace_index () 1;
Remove seed_can from SeedList;
for 1 to 8 do
if subspace_index () = 0
Find seed_can in subspace ;
// with the shortest distance to its vertex .
Push seed_can into SeedList;
Sort seed_can;
// according to pixel number classified in the subspaces.
return SeedList;

Table 2 shows an example of a seed list generated by IniSeedList from the color image, . Columns 1 and 2 in Table 2 indicate the defined subspaces and the number of the classified pixels in the subspaces, column 3 shows the order of seed_can in the list, and columns 4 and 5 list the RGB color components and locations of seed_can in , respectively. The top seed in Table 2 is seed_can = (244, 251, 243) with location (269, 271) from the subspace ν_white holding 100,369 pixels. After the algorithm pops out this seed for creating , UpdateSeedList produces a new seed candidate in the subspace ν_white from and inserts it on the top of the list. The new seed_can = (240, 251, 247) is located at (283, 267) in and is also from the subspace ν_white with 105,045 pixels. Please note that the input image to IniSeedList is , instead of itself. is an image extracted by that is the center of the RGB space. It is difficult to determine in which subspace the center color is classified, so any seed close to the center may not be able to differentiate colors very well, as shown in Figure 4(a). In order to guarantee to get a “good” seed, our approach does not select any seed falling in . It is also possible that there is no pixel in a subspace; that is, the seed candidates are less than eight. For example, the seed candidates for are not available in the subspaces ν_blue, ν_yellow, ν_green, and ν_pink.

SubspaceNumber of pixelsseed_can orderseed_can colorsseed_can locations

_white1003691(244, 251, 243)(269, 271)
_black193932(0, 6, 0)(193, 233)
_cyan703(151, 100, 57)(216, 190)
_red264(206, 166, 107)(225, 204)

The pseudocode of the CreateSubImage procedures is listed as shown in Pseudocode 3.

CreateSubImage (, center)
0; ø; ;
If (center = true)
FuzExtractor (, );
else ø;
SeedList IniSeedList ();
while (SeedList ø) do
The first element in SeedList;
FuzExtractor (, );
if ( & () )
– ( );
SeedList IniSeedList ();
return ;

CreateSubImage creates a set of subimages from a color source image. CreateSubImage invokes mainly three procedures: IniSeedList, UpdateSeedList, and FuzExtractor. The pseudocode of the FuzExtractor is shown in Pseudocode 4.

FuzExtractor (, )
Move into ;
Push four neighbor pixels of into queue;
while () do
the top element in queue;
Calculate using (3);
if ()
Move into ;
Push unduplicated neighbors of into queue;
return ;

CreateSubImage has two arguments: and center, where is a source image and center is a Boolean variable. If center = true, is created by , otherwise is assigned by the empty image ø. FuzExtractor is mainly implemented based on (3) and extracts a subimage . The FuzExtractor procedure starts with to generate with region growing. Because belongs to its subimage , FuzExtractor pushes its four neighbor pixels into a queue and checks the first-in element, , in the queue to see if it belongs to the subimage . If belongs to , its new four neighbors are pushed into the queue. The FuzExtractor procedure repeats this process until the queue is empty, which indicates that there are no more pixels in the source image matching the color. After FuzExtractor returns , CreateSubImage assigns to as described.

CreateSubImage uses two conditions to check if is significant or not. The first condition is to check that it is a ratio of area over one. If this ratio is too small, CreateSubImage ignores as holds too few pixels extracted from the source image . is most likely affected by noise and is not significant for any object in the source image. CreateSubImage also uses the condition, () & (), to check , where () is a variance defined byA big indicates a large area size of and a big () implies that contains the color components with a wide range distribution, as shown in Figure 4(b). Such a subimage is also considered as insignificant. If is significant, CreateSubImage assigns to a subimage and creates an , where holds the color components matching the color of and holds unextracted pixels in the source image. CreateSubImage iteratively processes until ø; that is, each pixel in the image is processed.

There may exist tiny holes in the subimage . Because these tiny fragments may be insignificant for object pattern features, the algorithm removes them by filling up such tiny holes with their original pixels. Usually, these tiny fragments are caused by a small RGB distance of two pixels that have the same RGB values and are located in the same row or in the same column, so the algorithm checks such a distance if it is less than a threshold, δ = 7. If so, the algorithm fills the tiny holes when creating the subimage, . The algorithm generates a seed from the subimage but always creates a subimage from the source image so it is very probable that the algorithm extracts an individual pixel into multiple subimages. We use a list to maintain according to a sending order of their sizes. Table 3 shows an example of the proposed algorithm that creates 20 subimages from the source image in Figure 5(a). Column 1 in Table 3 contains ID, columns 2 and 3 list the locations and the color components of the seeds for extracting the subimages using the FCE, column 4 shows the number of pixels extracted into the subimages , and column 5 records the variance of .

IDLocationsRGB colorsPixelsVariance

(269, 271)(244, 251, 243)14308148
(283, 267)(240, 251, 247)9674114
(267, 243)(230, 252, 239)64924150
(193, 233)(0, 6, 0)15643917
(358, 174)(228, 246, 232)10492157
(276, 280)(229, 232, 221)15307240
(292, 302)(209, 222, 213)10894372
(58, 141)(197, 216, 212)13815590
(292, 296)(1, 12, 4)30031419
(290, 302)(11, 14, 5)35291602
(464, 285)(7, 22, 25)39411591
(275, 252)(17, 30, 13)176451311
(320, 261)(20, 38, 26)48311961
(68, 138)(54, 39, 16)260092416
(216, 190)(151, 100, 57)29162394
(357, 158)(50, 53, 46)267692703
(293, 294)(185, 198, 189)11018534
(352, 171)(206, 191, 162)67427375
(49, 238)(89, 72, 42)285103311
(1, 320)(58, 86, 108)268454777

5. Merging Subimages

The proposed algorithm merges significant subimages into a set of meaningful segmented objects by the procedure MergeSubImage. The pseudocode of the MergeSubImage procedure is shown in Pseudocode 5.

MergeSubImage ()
for to do
for to do
if ;
Remove from the list;
for to do
return ;

The inputs to the procedure MergeSubImage are the subimages , and the output from MergeSubImage is the segmented image . Figures 5(a)-5(b) show a color image, , and the segmented image in the RGB space, , which is a union of 6 objects, ,  , shown in Figures 5(f)5(k). Figure 5(e) shows the manually made benchmark, .

MergeSubImage merges the subimages according to the overlap sizes of their color components to construct some objects in the source image. In the process of the MergeSubImage, the FCE usually classifies an individual pixel into multiple subimages. It causes these subimages to share a common color area in the RGB space, and even probably a subimage may fully cover another subimage. The common area of two subimages implies two features: spatial proximity and color similarity, because the two subimages connect together and have a partition with the same color if they share the common area. Therefore, our algorithm uses this unique characteristic to merge subimages. MergeSubImage uses the union to check the common area size of two subimages and in the RGB space. A bigger value of ()/ means that and share a larger common area; that is, their color components are more similar. A data structure for maintaining the subimages is a list that sorts in a sending order of their sizes. Technical implementation of merging is shown in Figure 6. The merging process starts with the last element with , located at the list tail, which is the subimage with the smallest area size, and checks with each subimage via . If , the algorithm merges and together, removes from the list, and replaces with , where size is bigger than one. After processing all the subimages in the list, the list holds the partitioned objects from the input source image . MergeSubImage returns the partitioned objects as well as its union, , as the segmented image .

Table 4 lists the extracted subimages from the source image in Figure 5(a). MergeSub- Image merges these 20 subimages into 6 partitioned objects in Figures 5(f)5(k). Table 4 also shows information on which and how many subimages are merged into which object. For example, the partitioned object shown in Figure 5(f) comes from a single subimage , while the partitioned object in Figure 5(j) is produced by merging 11 subimages , , , , , , , , , , . Figure 5(b) shows the segmented image by a union operation of .

Segmented objectsSub-images

, ,
, , , , , ,
, , , ,

6. Discussions and Conclusions

In this section, we present our selection of the membership functions’ parameters of the FCE defined in (4)–(7) and settings of the CreateSubImage and MergeSubImage procedures used for our tests. We empirically selected the parameters of the membership functions as , , , , and . A smaller α1 may more precisely segment color components in an input image but produces more extracted subimages. A slight change of does not affect the segmentation result, so it is not sensitive. should be at least less than 127.5. The threshold, , is important for classifying pixels into a subimage or an subimage . We select the parameters in CreateSubImage and MergeSubImage as , , , and . The function of is to remove insignificant subimages with a “too small” area that may be caused by noise. Both parameters and are defined to remove another type of insignificant subimages, for example, shown in Figure 4(b). This subimage is not extracted very well due to a “bad seed,” . Such a subimage is featured by its large area size and a large variance of its color components. The large variance indicates that the subimage holds a wide range of color distribution in the RGB space, so this subimage should be removed. is the threshold to check the overlap area over the subimage with a smaller area. In this paper, herein means that the overlap area is 60% of the subimage with the smaller area. The bigger is, the more similar the color components of the two subimages are. Changing affects the number of partitioned objects generated by MergeSubImage. As an extreme example, means that only if the larger subimage fully overlaps the smaller subimage, the two subimages will be merged together.

In our implementation, the algorithm invokes the CreateSubImage procedure twice. For the first invocation, an input image, , to IniSeedList and UpdateSeedList, is , where is extracted by . Because the first call is to process a source image , the algorithm does not select a seed close to the RGB space center. As discussed in Section 4, the seed cannot differentiate color components which are far from the well-defined eight colors. For example, the extracted image shown in Figure 4(a) holds many color components of six corn cobs. Because the algorithm does not select any seed from , it is very possible that some of the pixels in the source image remain unclassified. In order to process the unclassified pixels, we create an image , as shown in Figure 4(c). The algorithm calls the CreateSubImage a second time to process , but the input image to IniSeedList and UpdateSeedList is itself. The variance of is smaller than that of its original source image as most pixels with a large range of color distribution are classified after first calling CreateSubImage. For this reason, we halve the parameter of the membership functions to process . The proposed approach segments the color image in Figure 7(a) into 12 objects. Figure 7(b) shows the segmented image. Figures 7(f)7(q) display the first 12 partitioned objects. and in Figures 7(h) and 7(k) are extracted from in Figure 4(c).

In order to test the robustness of the proposed algorithm, we applied the algorithm with the same settings defined in the first paragraph of this section to process the five color images , in the RGB color space as shown in Figures 5(a), 7(a), 8(a), 9(a), and 10(a), respectively. Their corresponding segmented images are displayed in Figures 5(b), 7(b), 8(b), 9(b), and 10(b), respectively. The first four color images are from the database at UC Berkeley [23]. These selected color images represent different features and diverse scenes that might be taken under different light conditions. The source image in Figure 5(a) holds a bird on a tree trunk under the sky. The source image in Figure 7(a) holds six corns with many small multicolor kernels and its segmented image holding the six colorful corns. The source image in Figure 8(a) shows a lake and a mountain under the sky at evening. Figures 9(a) and 10(a) show the color images and . Table 5 lists the subimages, and the partitioned objects from the 5 color images. 85 subimages are extracted from , but the partitioned objects are only 6.



There is a lack of a strong mathematical model for precisely expressing the human’s understanding of colors due to psychophysical perception. For a given color, different observers may perceive its color definition differently. That is why the FCE-based approach classifying an individual pixel into several subimages is reasonable. The proposed approach to color image segmentation is easy to use, and it does not apply any procedure where one needs to preprocess source images. Removing subimages with “too small” areas is filtering short edges and noise in a color image. The approach automatically partitions a color image into an appropriate number of meaningful objects or regions. The manually made benchmarks are downloaded from the database at UC Berkeley [23], as shown in Figures 5(e), 7(e), 8(e), 9(e), 11(e), and 12(e), respectively. These benchmarks display desired segmentation results represented by the images’ boundaries. Although the proposed approach does not reach the benchmarks’ results, it is able to partition the color images into meaningful objects, for example, the segmentation results as shown in Figure 7.