Abstract
Maximally stable extremal regions (MSER) is a state-of-the-art method in local feature detection. However, this method is sensitive to blurring because, in blurred images, the intensity values in region boundary will vary more slowly, and this will undermine the stability criterion that the MSER relies on. In this paper, we propose a method to improve MSER, making it more robust to image blurring. To find back the regions missed by MSER in the blurred image, we utilize the fact that the entropy of probability distribution function of intensity values increases rapidly when the local region expands across the boundary, while the entropy in the central part remains small. We use the entropy averaged by the regional area as a measure to reestimate regions missed by MSER. Experiments show that, when dealing with blurred images, the proposed method has better performance than the original MSER, with little extra computational effort.
1. Introduction
Maximally stable extremal regions (MSER) [1] is a popular local invariant feature detection method because it has many highly desirable properties, such as invariance to monotonic intensity transformation, invariance to adjacency preserving (continuous) transformation, and low computational complexity [2–4]. It is shown in a detailed study [5] that MSER has the best repeatability and accuracy, except when dealing with blurred image. For blurred images, some local regions may no longer be detected by MSER, because MSER only detects regions whose shape is globally stable with respect to intensity perturbation. That is to say, the intensity difference should be large between pixels lying on the boundary and the outside. In the blurred image, sharp intensity contrast may be transformed into gradual variation. As demonstrated on the top of Figure 1, with MSER, some regions are lost in the blurred image due to the lack of sharp boundary, like the mouth and tail of the bird.
A blurred image can be seen as an image in a larger scale [6, 7]. Local feature detectors, which are good at dealing with blurring, like the Hessian affine detector and edge-based regions method, all rely on automatic scale selection methods. They construct a measure of local structure within the region and select the local maxima over the scales as the characteristic scale of the region. Even when the image is blurred, proper integration scale and differential scale can still be selected, the same structure can be found, not depending on intensity difference to find the boundary. A multiscale approach is used in [8] to improve MSER’s ability to detect features in blurred images, which is a close concept to automatic scale selection. A shortcoming in both methods is that they need to consider many scales in each image position, which greatly increases the computational complexity [9–11]. Moreover, when measuring the structure within the image, a further scale selection may be needed to compute derivatives, such as the Laplacian operator in the Hessian affine detector. Another way to make MSER workable on blurred image is to relax its globally stable criteria, as done in [12]. Stability is required only locally, for the primitives constituting the three-point frames. Higher number of features and better covered image are achieved, at the expense of more computational effort.
To enhance MSER’s ability in the situation of blurring, we can still borrow the idea of using local image structure as a measure to select the proper region scale. In a blurred image, considering MSER’s discarded extremal regions which are retained in original image before blurring, we can find a common phenomenon—intensities on the boundary vary more slowly compared to the original image, but the intensities in the central part are kept almost unchanged. In other words, entropy is large on the boundary and small inside. These extremal regions are lost because they are considered not stable with respect to intensity perturbation. To find these regions, we just need to find the extremal regions with minimal entropy. An advantage using entropy is that it does not rely on intensity difference, which may be changed in a blurred image. Another advantage is that entropy measures the region as a whole, instead of just one or a few pixels, which may be easily affected by blurring.
We just use the nested extremal regions extracted in the process of MSER for comparison of entropy, so that there is no additional effort for constructing series of regions of different scales, as what is done in automatic scale selection. Utilizing the nested extremal regions also facilitates the computation of entropy, which will be detailed in Section 2.
In the proposed method, we aim at finding the largest region with the minimal entropy. We use average entropy as our measure and find the region with the lowest average entropy within a local range in the nested extremal regions.
There is another popular local feature detector, “salient region” [13], using the entropy as a criterion. Different from our method, salient region looks for the smallest regions with maximal entropy, because it believes that these regions may contain more information. However, considering blurring, largest region with minimal entropy may be more proper. This is because, after blurring, entropy of region is inclined to increase. Therefore, region scale with maximal entropy will increase but scale with minimal entropy will decrease [14–17]. This could make the salient regions confusable with the background, but regions with minimal entropy are unaffected.
2. Entropy-Based MSER
The proposed method is an extension of the original MSER method, adding entropy as another criterion to find back extremal regions missed by MSER due to blurring. Here is the whole algorithm. Steps (1)–(4) describe the original MSER method, and step (5) is our additional step.(1)Sort all the pixels by intensity.(2)Place pixel one by one (in intensity order) in the image, and update the connected component structure, which forms the nested extremal regions.(3)Compute the area variation for each extremal region: Here represents the extremal region with maximal intensity , and refers to the extremal region expanded from the th one, with maximal intensity . is the relative difference of area when maximal intensity rises from to .(4)Traverse the nested extremal regions. Find the maximally stable extremal one, which has “var” smaller than its immediate parent and ancestor in the nest.(5)For the extremal regions not considered as maximally stable, compute their entropy averaged by area. If an extremal region’s average entropy is a local minimal in the neighborhood of the nest, we will select it as a complement to the MSERs.
It should be noted that the entropy of an extremal region can be easily calculated, because extremal regions are formed by sorted intensity; thus only subset of intensity is under consideration for each region. Here we show how to compute the average entropy in an incremental way.
Considering as an extremal region with maximal intensity and its direct parent in the nest . ’s intensity histogram can be represented as . Because contains , besides , there is additional intensity histogram . Entropy and area can be calculated as follows.(1)Area of : .(2)Area of : .(3)Entropy of : .(4)Entropy of : .
We can see that, with intensity growing up, maximal intensity, area, and entropy all increase. Considering a simple case which often occurs, , which means that the parent region does not contain other subregions except . Then there is a simple relationship between the average entropy:
It is easy to see that we can infer from in a simple way.
3. Experiments
In order to demonstrate the repeatability and time performance of our method when dealing with blurring, we compare it with the original MSER method, using open-source code provided at [18]. We use the 6 blurred images (bike set) provided by the popular Oxford dataset [19] for test. This blurred sequence is acquired by varying the camera focus [20, 21].
3.1. Repeatability Test
Here we show the results on the images for better comparison, as in Figure 2. We can see from Figure 2 that the main detection difficulties lie on the part where the intensity of the foreground is similar to the background, like the parts of the bike, and blurring makes things worse. Results show that our entropy-based MSER could successfully detect this kind of regions, which will consequentially improve detection repeatability. Furthermore, the number of detected features is greatly increased, which is desirable in the matching and recognition stages.
3.2. Time Test
It is also useful to compare the time consumption between the original method and our method. Table 1 shows the time consumption on the above bike set on a 2.4 GHz Pentium 2 CPU. We can see that no significant extra time is needed in our method.
4. Conclusion
In this paper we have proposed an efficient extension to the maximally stable extremal regions (MSER) which makes it more robust to blurring. We utilize the fact that, in a blurred image, the entropy of probability distribution function of intensity values increases rapidly when the local region expands across the boundary, while the entropy in the central part keeps small, and use entropy averaged by region area as a measure to reestimate region missed by MSER. Experiments show that when dealing with blurred images, our method has better performance than the original MSER, with little extra computational effort.
Acknowledgments
This work has been funded by the National Key Technology R&D Program in the 11th Five Year Plan of China under the program of Research on Disabled People Function Rehabilitation Aids (Grant no. 2009BAI71B07), the Research Project of Department of Education of Zhejiang Province (no. Y201018160), and the State Scholarship Fund from China Scholarship Council (no. 2011833105).