1. Introduction

It has been well demonstrated in numerous reports on physiology of vision (e.g., [1, 2]) that, in general, humans perceive known objects as collections of local visual saliencies. Several theories differently explain details of the process (see the critical survey in [3]), but there is a common understanding that when a sufficient number of local features found in the observed image consistently match correspondingly similar features in a known object, the object would be recognized. Although optical illusions may happen in some cases, such a mechanism allows visual detection of known objects under various degrading conditions (occlusions, cluttered scenes, partial visibility due to poor illumination, etc.).

Even without the psychophysiological justification, low-level local features have been used in computer vision since the 1980s. Initially, they were primarily considered a mechanism for stereovision and motion tracking (e.g., [4, 5]) but later, the same approach was found useful for many other applications of machine vision (e.g., image matching, detection of partially hidden objects, visual information retrieval). Typical detectors of low-level local features are derived from differential properties of image intensities or colors. The most popular detectors (e.g., Harris-Plessey [5] or SIFT [6]) are based on derivatives in spatial and/or scale domains and they do not retrieve any structural information from the image (even though, Harris-Plessey is often called a “corner detector”). However, there is a documented need for matching based on the local visual contents. For example, Mikolajczyk and Schmid in [7] presented cases of corresponding local features that are correctly detected but cannot be matched because of inadequate descriptors. Those features would be easily matched if the “visual similarity” between extracted patches can be quantified.

One of the most popular methods of image content matching is based on moment invariants which exist for various types of geometric and photometric distortions (e.g., [8, 9]). Several works employ them as descriptors of local features (e.g., [9, 10]) computed over circular windows (or windows of other regular shapes). Many alternative techniques for the local image content description exist as well. For example, local contrast measures have been reported (see [11]) as powerful descriptors in textured images. Another method, based on locally applied Radon filters, has been successfully used for description and recognition of human faces (see [12]). The above-mentioned approaches assume that local features can be matched by extracting (and comparing) properties invariant under distortions present in the analyzed images. However, the actual concept of “visual similarity” goes beyond that.

According to Biederman (see [1]), humans recognize known objects by identifying certain classes of geometric patterns that are combinations of contour and region properties. Such patterns may have diversified shapes, but all instances of the same pattern have the same structural composition that can be parameterized (at least approximately) using several configuration and intensity/color parameters. The method discussed in our paper follows this idea (although we do not use geons proposed by Biederman). The main assumption is that visual saliencies (local features) of interest correspond to various local geometric patterns that may exist within analyzed images. Even if the image is noised or distorted, the patterns (if prominent enough) should remain visible, although their appearances may be corrupted.

As in the majority of local feature detectors, the proposed method employs a scanning window of a regular shape. For rotational invariance, circular windows are proposed, but the method can work using windows of other shapes as well (e.g., squares or hexagons). Generally, the windows are larger in other detectors (because more complex contents have to be identified within windows), but the actual size of scanning windows is of secondary importance (as explained in Section 4). The objective is to detect those locations of the scanning window, where the window content is “visually similar” to a pattern of interest and to find the best approximation of the window by this pattern, that is, to create an idealized local model of the image. Two simple examples are shown in Figure 1, where digital circular windows of 30-pixel radius are approximated by a corner and a T-junction (the patterns that can be clearly visible in the windows).

Figure 1: Exemplary approximations of circular windows by patterns accurately corresponding to the actual visual contents of the windows.

Such locally found approximations can be potentially very powerful features for identifying similar fragments in images, for detecting partially visible known objects, for visual information retrieval, and for other similar tasks.

This paper presents analysis, discussion, and exemplary results on how such approximation-based local features can be defined and detected in images. Although certain aspects of the presented method have been already published (e.g., [13, 14]), this is an attempt to summarize the results and to highlight the identified advantages and drawbacks. In particular, the following issues are explored:(1) building accurate pattern-based approximations in the presence of degrading effects (techniques based on area moments and on modified Hough transform are discussed in Section 2);(2) quantitative methods of estimating “visual similarity” between approximations and the approximated windows (both indirect approaches, moment similarities, similarities based on Hough transform, and direct methods, Radon transform and image correlation, are briefly overviewed in Section 3);(3) definition, accurate localization, and scale invariance of approximation-based features (based on results of 1 and 2) are discussed in Section 4. In all sections, exemplary figures are used to illustrate the discussed effects and properties.

Preliminary concepts on how such approximation-based local features can be incorporated into image matching systems are briefly discussed only in Section 5 that concludes the paper.

2. Pattern-Based Approximations of Circular Patches

We assume that patterns of interest are defined by circular patches containing certain geometric structures. Patches of other regular shapes (e.g., squares, hexagons, etc.) can be considered as well, but circular patches are more universal because of their rotational invariance. Several examples of patterns of interest are given in Figure 2.

Figure 2: Exemplary types of patterns. Configuration parameters () and intensity/color parameters () are indicated for each pattern.

As shown in the figure, patterns are defined over circles of an arbitrary radius , and each instance of a pattern is represented (within the general characteristic of the pattern) by several configuration parameters (defining its geometry) and several intensities (or colors) describing the pattern’s visual appearance. The number of parameters (i.e., the complexity of patterns) is not limited, but patterns with 2–4 configurations parameters (and similar numbers of intensities/colors) are the most realistic ones for scanning windows of a limited diameter. All examples shown in Figure 2 are such patterns. For example, a T-junction pattern is defined by three colors , , and , the angular width , and the orientation angle .

When an image is analyzed, we attempt to approximate contents of a scanning window by the available patterns. The pattern-based local features are found at locations where “the best approximations” exist. Parameters of those approximations would be used as descriptors of the features. In our researches, the radius of scanning windows ranges between 7 and 25 pixels. Smaller windows do not provide enough resolution for patterns with fine details, while larger windows unnecessarily increase computational costs.

Formally, the pattern-based approximation consists in computing the optimum configuration parameters and intensities/colors for a given content of the scanning circular window. Knowing the optimum parameters, we can synthesize the pictorial form of the approximation (as shown in Figure 1). The synthesized images are used mainly for visualization (to estimate how accurately, from the human perspective, the original image has been approximated) and, generally, are not needed for other purposes.

2.1. Moment-Based Approximations

Our previous papers (e.g., [13]) presented a moment-based technique for producing approximations for various patterns. It was based on the observation that configuration and intensity/color parameters of patterns can be expressed as functions of low-order moments computed over the whole circle. For example, the angular width of a corner pattern (i.e., one of its configuration parameters, see Figure 2) is equal to while the orientation angle for a T-junction pattern (see Figure 2) satisfies where are moments of order computed within the system of coordinates placed in the window center.

Intensities of the approximations can be also expressed using moments. For example, three intensities of a T-junction pattern (see Figure 2) satisfy the following system of linear equations: where and indicate and , correspondingly.

Alternatively, when the configuration parameters are already known, we can estimate the intensities/colors of the approximations by averaging intensities/colors of the corresponding regions within the approximated patch.

Equations (1)–(3) (and their counterparts for other patterns) are basically the same for both grey-level and color images. The only difference is that for color images, moments are 3D vectors (moments computed for RGB components), so that the expressions should be modified accordingly (details are discussed in [15]).

The expressions derived for a certain pattern can be applied to a circular image of any content, and the obtained values (if the solutions exist, e.g., (1) or (2) may not have any solution) become parameters of the approximation of the given image by this pattern.

This method has several advantages. First, it produces accurate approximations even for textured images (where other techniques, e.g., the corner approximations discussed in [16], fail) and for heavily blurred patterns where visual identification of a pattern is difficult even for a human eye (see examples in Figure 3 for corner patterns).

Figure 3: Exemplary moment-based approximations for a corner pattern.

The method can also identify windows which cannot be approximated by the pattern of interest (the corresponding equations have no solutions). Exemplary circular images for which approximations cannot be found are given in Figure 4.

Figure 4: Circular images for which approximations do not exist for corner (2 examples), T-junction, and pierced round corner patterns, correspondingly.

There are also disadvantages of the moment-based approximation technique. First, in many cases, it produces an approximation even though the visual inspection clearly indicates that the window content is not similar to the given pattern. Several examples of such scenarios are given in Figure 5.

Figure 5: Visually incorrect approximations of circular images by corner, pierced round corner, and T-junction patterns.

Secondly, the quality of approximations may be strongly affected by “visual intrusions,” that is, unwanted additions to the image content caused by other objects, illumination effects, or just by the natural nonuniformity of images. A relatively mild effect of “visual intrusion” is shown in Figure 6(a), where a dark stripe affects accuracy of the corner approximation produced by the method. A much worse situation can be seen in Figure 6(b), where an external object enters the circular window and completely distorts the approximation by a T-junction pattern (even though, the shape of the actual junction within the image is not affected by the intrusion).

Figure 6: Examples of (a) a mild distortion and (b) strong distortion of the moment-based approximations caused by “visual intrusions.”

Moment-based approximations are also difficult mathematically. Equations for calculating approximations parameters (similar to (1)–(3)) should be individually designed for each type of patterns. Even for relatively simple patterns, polynomial expressions of higher orders are needed. For example, approximations by pierced round corner pattern (see Figure 2) use 4th-order polynomial equations. Moreover, the limited number of low-level moments (higher-level moments are too sensitive to noise and digitization effects) naturally limits the number of parameters, that is, the complexity of patterns. It is very difficult to find a reasonably simple solution for patterns with more than 3 configuration parameters (and the corresponding number of intensities/colors).

2.2. Approximations Based on Hough Transform

Patterns considered in this work can be represented as unions of grey-level/color regions and contours defining boundaries between those regions (see Figure 2). Thus, an alternative method of building pattern approximations can be based on contour detection techniques. Several similar attempts have been reported previously (e.g., [17]), but our objective is to develop a tool suitable for patterns more complex than typically considered corners or junctions.

We propose to use a well-known Hough transform with modifications addressing needs and constraints of the problem. First of all, the calculations are performed within the limited area of scanning windows so that, in order to provide enough data, all images pixels are involved instead of contour pixels only. This technique (preliminarily proposed in [18]) exploits directional properties of image gradients.

Assume that Hough transform is built for the family of 2D curves specified by equations with parameters .

Each pixel of image contributes to () accumulator in the parameter space the dot product of the image gradient and the unit vector normal to the hypothetical curve (both taken at coordinates): Thus, regardless the gradient magnitude, only the gradient components orthogonal to the expected curve are actually taken into account. For example, if concentric circles (or their arcs) are detected, only the radial components of the gradient are taken into account, while for detecting radial segments, only the components that are orthogonal to radials (see Figure 7).

Figure 7: (a) Exemplary intensity gradient and (b) its contribution to the Hough accumulator when detecting radial lines and (c) detecting concentric circles.

We additionally increase the contribution of pixels proportionally to their distance from the circle’s center because of poorer angular resolution in the central part of digital circles. A somehow similar problem has been handled in [19] by using polar coordinates.

After contours of a pattern-based approximation have been extracted, intensities/colors of the corresponding regions can be estimated using the methods described in Section 2.1.

There are several advantages of using Hough transform for building pattern-based approximations of circular images. In particular, the approximation results are generally much less sensitive to “visual intrusions.” Figure 8 shows examples where in spite of intrusions distorting the “idealized” contents of circular windows, the accuracy of approximations is very good, much better than by using moment expressions.

Figure 8: Comparison between moment-based approximations (top row) and approximation based on modified Hough transform (bottom row) in case of “visual intrusions.”

Moreover, approximations can be often obtained even if the pattern areas differ only in textures, while the average intensities/colors are identical. An illustrative example of such case (where corners can be hardly identified) is shown in Figure 9.

Figure 9: Examples of corners produced by texture differences only. The approximations have been accurately found based on Hough transform.

Another important advantage is that Hough transform-based approximations can be decomposed and built incrementally. In many cases, contours defining the pattern boundaries consist of fragments that can be detected (using Hough transform) separately. The configuration parameters of already found contour components can be used as default values for detection of subsequent fragments.

A pattern shown in Figure 10 (sharp pierced corner) has four configuration parameters (orientation , angular width , radius of the hole , and distance ). Search in a 4D parameter space would be computationally expensive. However, the corner component of the boundary can be identified using only a 2D space (the orientation angle and the angular width ). Given the orientation angle , the hole parameters can be found in another 2D space (radius and distance ).

Figure 10: A pattern with four configurations parameters. A 4D parameter space used for Hough transform-based approximation building can be decomposed into two 2D problems.

Some weaknesses of this method also exist. In particular, approximations built using Hough transform may have random, incorrect configurations in heavily blurred images. An example is given in Figure 11.

Figure 11: Corner-based approximations of a blurred image obtained using moments (left) and Hough transform (right).

It can be eventually concluded that techniques for building pattern-based approximations of patches can be based on both integral (moments) and gradient (Hough transform) properties of approximated images. However, gradient-based mechanisms should be considered the tool of primary importance.

In this paper, we discuss only relatively simple techniques with low computational complexity. Although more complex mathematical models have been proposed for the same or similar problems (e.g., [12, 17, 20], etc.), we believe that for the majority of intended applications, the methods discussed in this paper provide at least satisfactory solutions.

3. Accuracy of Approximations

The main objective of building pattern-based approximations of patches is to obtain robust local features, that is, features that can be reliably detected in images that are distorted and degraded by various effects. This assumption would be justified if the approximations are actually similar to the approximated fragments. However, as shown in Section 2 (e.g., Figures 5 and 6), visual appearances of approximations may strongly differ from the approximated images. Such approximations are obviously useless, as potential local feature, because the visual structures of the original images are lost.

Therefore, there is a need to quantify similarity between approximations and approximated patches. Only those image locations where the highest similarity exists between window contents and their approximations would be used as the local features of interest. The similarity measures should obviously correspond to the “visual similarity” (i.e., the similarity subjectively estimated by a human observer) between images. Additionally, the measures should be simple enough to be repetitively applied to the window scanning images.

The most straightforward similarity measure would be a cross-correlation which does not even need normalization because we expect roughly the same colors/intensities in circular patches and in their pattern-based approximations. However, as discussed in [13], neither the overall cross-correlation (i.e., computed over the whole patch) nor any combination of regional cross-correlations (i.e., computed separately for each region of the approximation) is a reliable measure. Figure 12 shows several circular patches and their corner approximations. Visually, all approximations are equally similar to the approximated patches, but the correlation-based similarities (given in Figure 12) are very different. Therefore, even though the features can be found as local maxima of the similarity values, the correspondence between the visual similarity and the similarity measure is very poor.

Figure 12: Examples of corner approximations of similar visual quality but different similarity measures (based on cross-correlation).

Moreover, to effectively use the cross-correlation as a similarity measure, the approximation images should be synthesized (with the resolution corresponding to the size of patches).

Thus, alternative similarity measures with lower computational complexity have been proposed and tested. Similarity of low-level moments and similarity of Radon transforms have been reported in [21, 22], correspondingly. They provide more uniform correspondence between “visual quality” of approximations and computed similarities (exemplary results showing a simultaneous deterioration of both “visual quality” and computed similarities are shown in Figure 13). These measures are not sensitive to (uniformly distributed) noise, so that their global maxima can be used to determine positions of the pattern-based local features.

Figure 13: Corner approximations of gradually deteriorating both “visual quality” and computed similarity (similarity measure based on low-order moments).

It should be noticed, however, that in Figure 13, the similarity values change very slowly, much slower than the visual similarity that deteriorates rapidly. This is a significant disadvantage of such measures, as further discussed in Section 4.

Moreover, all abovementioned similarity measures are very sensitive to visual intrusions, so that even accurate approximations (e.g., built using Hough transform) may not be recognized as such.

An entirely different similarity measure can be proposed if Hough transform is used for building pattern-based approximations. For accurate approximations, the content of the winning bin in the parameter space is usually a prominent spike, while for less accurate approximations, the spike is less protruding. Thus, after testing several other approaches also based on Hough transform, we propose the similarity measure as the ratio of the winning bin height over the sum of all bins’ contents. Exemplary results given in Figure 14 show how significantly this ratio changes when the scanning window moves away from the actual pattern location. In this example, T-junction pattern has been deliberately selected because it needs only a 1D parameter space.

Figure 14: Three locations of the scanning window and the corresponding parameter space values (bin contents) for Hough transform of T-junction pattern. The central column shows the window at the position matching the actual junction.

Currently, we consider this measure of similarity superior to other tested approaches, as far as the feature localization is concerned. However, this is not an absolute measure, that is, its values fluctuate significantly when the image is noised, even if the noise neither affects the “visual quality” of the pattern nor modifies the produced approximations. A self-explaining example (with the approximations superimposed over the original images) is shown in Figure 15. Thus, localization of the pattern-based features should be again based on detecting local maxima of similarity.

Figure 15: Changes of the bin contents for Hough transform of T-junction pattern caused by a high-frequency noise added. The original results are shown for the reference.

In the future applications, we plan a combination of this similarity measure with secondary area-based measures (Radon transform or moments). The primary measure would be used to localize the feature candidates. The secondary measure would provide the (absolute-value) estimate of whether the local maximum of the primary measure is actually a high-quality approximation or whether it should be ignored.

We can, thus, conclude that while accurate pattern-based approximations can be found relatively easily, it is more difficult to quantify the accuracy of approximations in a manner corresponding to a visual assessment by human observers. Measures are needed that (1) produce similarities proportional to the “visual quality” of approximations, as perceived by humans, (2) are insensitive to noises degrading the overall quality of images, (3) are robust against visual intrusions that do not affect the actual patterns of interest, and (4) produce sharp maxima for the actual locations of the patterns. The existing measures are not fully satisfactory yet, and we believe that a further development of similarity measures is an interesting topic of practical importance.

4. Approximation-Based Local Features

4.1. Detection and Localization

Based on the explanations given in Sections 2 and 3, the definition of approximation-based local features is straightforward.

A local feature (of radius ) defined by pattern exists at a location within the analyzed image if: (1)the approximation by pattern of the circular window of radius located at exists;(2)similarity between the approximation and the window content reaches a local maximum at ;(3)(optional) the value of the absolute similarity measure (see Section 3) exceeds a predefined threshold. Configuration and intensity/color parameters of the approximation are considered descriptors of the feature.

In practice, implementation details of the above definition can vary. For example, it is well known that standard keypoint detectors (e.g., Harris-Plessey or SIFT) produce significant numbers of keypoints in typical images of natural quality. It is, therefore, possible to select the keypoints produced by such detectors as preliminary candidates for approximation-based local features and apply the method only to these locations. The advantage of such an approach is that the only task is to build the approximations and to estimate their accuracy (the localization of feature candidates is performed by the keypoint detector). Another recommended option is to scan images using larger position increments and to conduct a pixel-by-pixel search only around locations where approximations are found with a reasonable accuracy.

It should be noted, nevertheless, that both in the original method and in its improved variants, the same location can produce several pattern-based features. This happens if the window content can be approximated with a comparatively similar accuracy by several patterns.

Unless feature candidates are prelocated by an external keypoint detector, the similarity values are used to localize the approximation-based features. Unfortunately, as indicated in Section 3, the area-based similarity measures (i.e., cross-correlation, moments, and Radon transforms) do not perform well in this problem. Even in high-quality images, there is a tendency to detect clusters of pixels with comparable similarity values instead of producing sharp maxima. The actual location of the feature would be somewhere within a cluster, but the similarity variations are so small (see the example in Figure 13) that a minor noise, a small distortion, or even digitization effects may shift the maximum of similarity to a distant part of a cluster. Figure 16 shows clusters produced by corner approximations for an exemplary image of perfect (digitally) quality. Note highly uniform similarities (represented by intensities) within the clusters.

Figure 16: Localization problems for corner features using area-based similarity measures.

However, similarity measures based on Hough transform localize features with pixel accuracy (we do not consider subpixel accuracy although certain possibilities are discussed in [8]). Exemplary results for two corners from Figure 16 are given in Figure 17 (similarities are again represented by intensity levels). Figure 18 shows an exemplary image and several pattern-based features detected within this image.

Figure 17: Localization of selected corner features using the similarity measure based on Hough transform.

Figure 18: A image and several approximation-based local features detected (shown in three images for better visibility).

4.2. Are Approximation-Based Features Scale-Invariant?

Approximations discussed in this paper are built over circular images of radius . Therefore, in principle, the method is not scale invariant. Any change of radius (or image rescaling) may result in different sets, different descriptors, and/or different localization of approximation-based features.

However, from the practical perspective, the proposed features should be considered scale invariant within a certain range of scales. Figure 19 shows an exemplary image with several approximations obtained for windows of two significantly different diameters. The results given in Figure 19 illustrate a more general property of approximation-based features. As long as the scanning windows are large enough to include the approximating patterns but small enough so that the patterns are not visually suppressed by prominent features from the neighboring areas, the size of scanning window is actually not important for detecting pattern-based features. Of course there are certain limits but we conclude from the preliminary experiments that for typical images, the radius of scanning windows can vary within approximately 50–200% range without significant changes in the results. Most of detected approximation-based features are the same, and their characteristics (parameters of approximations) also remain unaffected.

Figure 19: Exemplary pattern-based local features obtained by using scanning windows of significantly different sizes.

Because the numbers of approximation-based features extracted from a single image are rather large (depending primarily on the image complexity and the number of available patterns), many of the features become effectively scale invariant in the sense explained above.

5. Summary

Currently, the most prospective area of application for approximation-based feature is visual information retrieval (VIR). Although computations used in the proposed algorithms are simple, the amount of data to be processed (moments and/or Hough transforms calculated over scanning windows of significant sizes applied to large images, determining similarities between approximations, and windows, etc.) is prohibitively large for typical real-time tasks (e.g., for vision-based search operations in exploratory robotics). Thus, the advantages of approximation-based features reflect primarily our VIR experiences and goals.

We envisage that database images will be preprocessed, that is, approximation-based features are predetected and memorized in the database together with the images. Such feature detection and memorization for all database images can be done offline whenever computational resources are available. The additional memory requirements are insignificantly small compared to the memory needed to store the images themselves. New types of approximation-based features can be incrementally added to the databases when approximation builders for new patterns become available.

The proposed features are a natural candidate for matching images since they provide local visual semantics of the analyzed images. Whenever a query image is submitted, it would be processed in the same way. Subsequently, local feature extracted from the query image would be matched against the database features. If enough evidence is found that the local semantics of the query image and of a database image are similar (e.g., approximations by the same patterns are extracted at correspondingly matching locations and descriptors of the approximations are correspondingly consistent), the images may contain visually similar fragments. Because the configuration descriptors of the features are considered more significant than colors/intensities, images containing visually similar fragments can be matched even if they are seen in completely different visual conditions (nonuniform changes of illuminations, different coloring, etc.). Nevertheless, variations of the matching algorithm are possible (depending on the applications), so that colors/intensities can be considered important descriptors as well.

Comparing to matching techniques based on other local features, the complexity of matching using the approximation-based features can be significantly reduced. Approximation-based features are categorized by approximating pattern so that only features approximated by the same patterns are the potential matches. Thus, the estimated number of attempted matches is reduced exponentially. Additionally, the method allows “targeted” image matching by using only a subset of available patterns (those representing the visual contents considered important in a given problem).

The issues of effective image matching using the approximation-based local features are not discussed in this paper. Generally, the techniques are similar to already known algorithms, for example, geometric hashing (see [23]) or methods used in [6, 7, 10].

The paper has presented only the principles of the proposed methods and approaches. Thus, no conclusive statistics on the method’s performances can be presented yet. Currently, the methods are integrated into a working platform that can be used for selected applications. One of the important issues is expansion of the list of available patterns so that complex images can be described by large numbers of more diversified features. It is our hope that the proposed approach can be developed into useful tools for visual data storage and retrieval systems (including internet browsers for visual contents). Further results of currently conducting researches will be addressed in future papers.

Acknowledgments

The results presented in the paper are done under STAR Science and Engineering Research Council Grant no. 072 134 0052. The financial support of SERC is gratefully acknowledged.