Abstract

This paper proposes one possibility to generalize the morphological operations (particularly, dilation, erosion, opening, and closing) to color images. First, properties of a desirable generalization are stated and a brief review is done on former approaches. Then, the method is explained, which is based on a total ordering of the colors in an image induced by its color histogram; this is valid for just one image and may present problems in smoothly coloured images. To solve these drawbacks a refinement consisting of smoothing the histogram and using a joint histogram of several images is presented. Results of applying the so-defined morphological operations on several sets of images are shown and discussed.

1. Introduction

Mathematical morphology is a well-established technique for image analysis, with solid mathematical foundations [1] that has found enormous applications in many areas, mainly image analysis, being the most comprehensive source the book of Serra [2]. A good modern introduction to mathematical morphology is provided in [3]. Most of the practical applications are based on clever combination of a small set of operations, namely, erosion and dilation, based in turn on the hit-miss transformation, or their direct derivatives, opening and closing. Mathematical morphology was initially developed for binary images and later on generalized to gray-valued images [2, 4], considered as a sampled function of    in , or in general of any function of in . Nevertheless, color (or in general, multispectral) images are samplings of functions of in , with being equal to three in the case of the usual color images or to the number of bands otherwise. A good generalization of morphological operations to these kinds of images would be an extremely useful and interesting goal, since it would extend the already tested techniques of image analysis based on morphology to color images. Such techniques include, amongst others, robust methods for shape analysis, border detection through morphological gradient, segmentation, and texture analysis. Therefore, this work is a proposal of one possible generalization of morphological operations to color images. Later we comment that we do not think a single natural generalization exists and this work does not intend to obtain such a thing; its foundation arise in an obvious extension of the concepts of “background” and “object,” widely used by the binary morphology.

The paper is organized as follows: Section 2 recalls the basic definitions of the morphological operations and the properties they should accomplish. Next, Section 3 introduces the state of the art in color morphology; Section 4 explains our approach in detail including the properties of this formulation and variants of the proposed method. Section 5 presents the results of applying the method to a selected set of representative color images and finally Section 6 draws conclusions and points to future work.

2. Requirements for Colour Morphological Operations

The first important thing to do is to establish the conditions that any reasonable generalization of morphological operations should fulfill. Let us remember that the two basic operations (erosion and dilation) applied to functions (gray level images) are defined as follows: let be a set in and let be a point called its origin. Let a gray scale image be an upper semicontinuous function of in with compact support and with values for each point such that , with being a positive real constant. Erosion of image by the structuring element denoted as is defined as another image whose value at any point is Similarly, dilation of image by denoted as is the image: with being the set translated to (with its origin on ). Compound operations called opening and closing are defined as where is the set that results from reflecting with respect to its origin. From these definitions it is clear that an order relationship must be defined in the set of values that can take so that a supremum and an infimum are defined for any finite set of points. Note that binary morphological operators can be considered as a particular case of the former definitions, as long as the value used for objects is bigger than that of the background. In the previously defined case of gray scale images taking values on that order is the usual order relationship between real numbers; if we generalize the concept of image to a function taking values in the order is not canonically defined, and this is the difficulty that all methods for generalising morphology to colour images must address.

The properties that a useful morphological operation has are derived by those of dilation and are summarised by Serra in [2]. They are as follows. (i)There should be an idea of ranking induced by an order. (Note that order, and no value, is sufficient. Actual values are not used in the morphological elementary operations, which are based on minimum and maximum values in given sets). (ii)For any finite set, the ranking should yield a supremum. (iii)We must have the possibility of admitting an infinity of operands (i.e., dilation and the other operations could be applied as many times as needed).

Apart from that, other desirable properties that a colour dilation should have are summarised by Köppen and coworkers in [5]: (i)The colour dilation should be colour preserving; that is, no new colours should be introduced (as in gray level: no new level appears in a dilated or eroded image). (ii)The colour dilation should be an increasing operation, that is, with being any image and any structuring element. This requires the definition of an order relationship between colour pixels, which is the key point of all this matter. (iii)If denotes standard binary dilation, the colour dilation should be compatible with it, that is, with and being structuring elements whatsoever. (iv)Restriction of the definition to gray scale images should become the standard gray level dilation.

3. Previous Work

Globally, there are two main possibilities to define morphological operations on color images: (1) consider the colours as labels associated to each pixel or (2) use the color values to establish a total ordering in the color space. Most of the approaches belong to the latter type. Indeed, the only one clearly of the first type is from Busch and Eberle [6].

On the other hand, applications that are based on an order relationship are the majority (including those proposed in this paper). All of them cope in different ways with the problem of ordering a vector space (in this case ). This has already been addressed by classical statistics (see [7]). In it four types of ordering are mentioned: marginal ordering (MO): only one component of the vector is used by relaying in the usual order in ; partial ordering (PO): groups of samples are considered and a given order is established amongst groups; conditional order (CO): order is established using a given set of the components, conditioned to the order by another set; reduced ordering (RO): a function from in is defined and the points are ordered by the value of this function, using the standard order in . The function must be injective. Only the third and fourth of this ordering criteria give place to an order relationship, for which exactly one of the relations , or can be established for any pair of vectors .

The most widely used is RO. Nevertheless, a prestated RO has an important drawback: in some cases it may be counterintuitive and this is a key point of our approach. To see this let us consider the example of an image with a uniform colored spot in the middle of a uniform dark background. Erosion with a disk-shaped structuring element larger than the spot should eliminate it, no matter which colors spot and background have; nevertheless, if a reduced ordering is used in which the color of the spot dominates over that of the background the spot would be enlarged. It is important to note that a similar counterexample can be found, irrespectively of the color space and the ordering function used.

Comer and Delp in [8] define erosion and dilation using both, MO and RO, and compare the results of applying the 2DCO filter of Stevenson and Arce [9] to synthetic noisy color images. Recently [10] introduced a new vectorial ordering based on MO to avoid introducing new color artifacts in the image. Another interesting approach is from Köppen et al. [5, 11] who use fuzzy sets to establish a total order in a set of vectors (colors). Along the same line, [12] proposed a new vector ordering scheme based on a fuzzy if-then rules system.

Angulo in [13] proposed another example of RO. He defined a total order relationship using distance to a reference color completed with lexicographical cascades. The idea was to choose a color space with each color a 3-component vector. Then, choose a metric in this vector space (Euclidean, Mahalanobis, or another) and finally a reference color . In [14], a lexicographical cascade was also used to complete total ordering after reducing vectors to scalar components with the help of a nonlinear weighted combination. A similar approach based on Lexicographical order operators and distance-based criteria was recently used in the work proposed by [15].

Finally, Hanbury and Serra in [16] used a quite complex ordering scheme based on a physical analogy that took into account the human side by using a perceptual system of color coordinates, namely, the . In this work, the authors took as the ordering function the potential created by a virtual arrangement of charges, each corresponding to a color present in the image, in the space. Each charge created a potential at a given point inversely proportional to the Euclidean distance between the charge and that point, measured in the space.

4. Description of the Approach

Our proposal is based on establishing a total order in the set of pixel values (colors) using a function , so it fits in the category of reduced orderings. As in former approaches, color morphology is transformed in gray level morphology in the set of values of so all desirable properties are preserved. Current RO approaches aim to impose a total order based exclusively on the colors themselves, and not on the image so they try to impose a canonical order to the set of colors. Moreover, they depend on the color space chosen.

The following subsections explain each of the variants of the proposed method which are successive refinements intended to correct the drawbacks of the previous variant. Essentially, they are based on using the histogram of the image to be processed to get the color ordering (Section 4.1), on using a smoothed version of that histogram (Section 4.2), and finally on using a compound histogram obtained from a class of similar images (Section 4.3).

4.1. Morphology Based on Plain Histogram

Unlike previous work, function is here calculated for each image and tailored to it. The problem of the computational cost of doing this will be addressed by working on groups of similar images. The problem is obviously how to choose a good function for a given image . The idea comes from binary morphology: erosion aims to reduce the objects substituting parts of them by the background. The opposite is true of dilation. But normally, the background is the largest region of the image. Therefore, we will define the background as the color that appears most frequently in an image. This will be the less important one and will be dominated by all others. In general, the less frequent a color is in an image, the more dominant it will be. This is equivalent to creating a color histogram of the image and assigning the highest value to the color that gives the smallest peak, a smaller value to the next peak, and so on. If two colors have exactly the same number of pixels (which is quite unlikely in real images) a random choice is done. Formally, let a color with three components in any color space, each of them quantized in the integer range . The color histogram of is a function of defined as Each bin of the histogram contains just one color; this histogram is a plain array with no reference to the color space used. With respect to the reassignment (the values), the actual values assigned are in fact irrelevant, since morphological operations rely only on order amongst values, and not on the values themselves. An appropriate choice is , where is the number of pixels in and . Function is an image defined at any point where is defined, and its values (which are integers) are in the range .

Once this auxiliary image has been built with the ordering values (-values) assigned to each pixel, morphology can be done on it as if it was a gray level image. Finally, colors are re-taken from the original assignment; there is no ambiguity in this process since the function is injective. The main advantage of that is that it does not depend on the color space used to represent the image. Indeed, colors are taken as mere labels. The results of this approach can be seen in Figure 1 (Section 5) where they will be discussed in more detail.

4.2. Morphology Based on Smoothed Histogram

The idea presented in the former subsection has one obvious problem: it can work extremely well for images with few colors, like maps, made by large planar patches of exactly the same tonality but it would not work as expected in real images constituted by continuous grading of tonalities and shades. For these images each specific color appears too infrequently; therefore, it is not appropriate to take its frequency as a significant measure of importance. This can be seen as a problem of lack of sufficient samples in a 2D image to build a meaningful histogram in the 3D color space.

The first solution we propose is to consider the 3D color histogram as the probability density of the appearance of colors in the image, and to smooth it so that each color exerts influence over neighbor colors. In this way a single color appearing as various tonalities or shadings would be ordered with respect to other complex colors according to its global importance. Later we address this problem by adding more similar images. In that way we have a larger number of samples. The smoothing kernel chosen is, by resemblance with Hanbury, a potential-like function. Each pixel of the image is considered as a unit charge in the color space, which will be quantized as stated before, and so it is a cube of . The potential created by pixel with color at any point of the color space will be where is a suitable distance in the chosen color space between points and and a smoothing parameter such that , to be chosen. In the experiments of Section 5 we test for two distances in each of the color spaces: Euclidean and Mahalanobis. Note that, since the color space is quantized taking one as the minimal distance between points, and so . Once all charges have been placed, potential at any point of the color space is calculated as A reasonable way to choose the parameter is to place a unit charge at the center of the admissible 3-D cube (or shape) of the color space and make its potential at the furthest border equal to just a negligible amount (1%) of the potential at the charge (which is 1). For our case the and spaces used were quantized to 100 parts in each component and so a reasonable value for was 2. Then the points in the color space are simply ordered by its potential . A possible problem is that the application defined by this potential might not be injective. The solution and some comments on the implementation can be seen in [17].

The method is still clearly of reduced ordering type. Now a dependency on the color space has been introduced, so the results shown in Section 5 have been tested with two color spaces: and .

4.3. Morphology for a Class of Similar Images

The obvious inconvenience of the former method is that it is necessary to calculate (which is computationally expensive) for each image. The solution we have used is to build the histogram not for a single image but for a set of similar ones, accumulating their pixels in a global 3-D histogram, appropriately weighted. The weight for each pixel is inversely proportional to the number of pixels of the image to which it belongs. This is done so as to give each image the same influence independently of its size. However, the weights can be changed if the user has additional information on the color distribution of each sample. This is like trying to estimate the color distribution of a class, as a mixture of those estimated from each image in the class and add robustness since more samples are used.

The process of calculating the potential is as in the former case and generates a 3-D array that we will call from now on the cube of potentials.

The idea behind this approach is that each type of images is characterized by the proportion in which the different colors appear in it. If this can be assumed and a sufficient number of samples are taken results should be similar to the single smoothed histogram method. This is shown in Section 5.

4.4. Properties

These definitions, either the use of histogram or the smoothed version in its two variants, fulfill many of the desirable properties of morphological operations.(i)There is total order (a supreme for each finite set).(ii)Operations can be applied iteratively. (iii)Erosion and dilation are color preserving (no new color is introduced). (iv)Dilation is increasing, according to the color ordering defined for that image and color dilation is compatible with the binary dilation, since it is a direct generalization of gray level, which it is.

Nevertheless, they do not accomplish these properties. (i)Restriction to gray level does not equal the standard gray level operation, since order by frequency of appearance is not a monotonic transformation.(ii)Erosion and dilation are no longer translation invariants unless the translated image wraps around the borders of the image (toroidal mode) since some pixels would be discarded and others would enter into the image field, altering the colour distribution and consequently the histogram-induced colour ordering; for the same reason, the local knowledge property is violated. The former is because the histogram of the image changes and the latter because local knowledge relays on the restriction operation (an image clipping) which also changes the pixel distribution.

Let be the definition domain of the image ; accomplishes the local knowledge property if there exists such that with being the restriction (concretely, ). But restriction gets rid of some pixels, which alters the histogram.

5. Results

A problem always present on color morphology works is the choice of an appropriate evaluation criteria. We believe that all the published results rely on subjective visual evaluation. This is because, even if a suitable metrics between color images (Euclidean, perceptual, etc.) exist, there is no canonical result with which the obtained images can be compared. We think this is a difficult problem for which we have no obvious solution so we are also obliged to use subjective evaluation.

The experiments consist of the application of the basic morphological operators (erosion, dilation, opening, and closing) with a disk-shaped structuring element on several images chosen to illustrate the advantages and inconveniences of the three variants of the proposed method. Also, an implementation of the method given by Angulo in [13] is shown as a comparison with a different proposal. Essentially, this is a method that establishes an ordering with respect to a reference color; let us call it . The ordering between two colors, and , is such that if and only if ; that is, colors are ordered inversely to their distances to the reference color. The distance he chooses is the Euclidean in RGB and spaces and a modification taking into account the angular nature of coordinate H for the LHS space. In the case that a lexicographic ordering by component is chosen, which consists of ordering using first G, then R then B for the RGB space and first , then , then for the space.

The first image is a map of the Valencian Community (Spain) with very few colors (32) and a very simple structure; the next is a collection of paintings from Joan Miró composed mostly by planar patches, but also containing thin lines and graded tonalities. The last example is a real image of a mosaic dragon (Güell Park, Barcelona) with several colors and a natural background similar to the one used by Hanbury and Serra [16]. All the experiments are done in two color spaces: and the perceptual space and with two distances to apply the histogram smoothing kernel: Euclidean and Mahalanobis. In the paper we only show the experiments for the Euclidean distance; those for Mahalanobis can be seen on the web page (http://johnford.uv.es/VISION/ColorMorphology/index.htm).

5.1. Results Obtained without Histogram Smoothing

Figure 1 shows the map of the Valencian community (a) together with its erosion (b), dilation (c), opening (d), and closing (e) with a disk-shaped structuring element of 3 pixel radius (note: the differences between the color images may be hard to see due to their size and to the deficient color quality of some printers. Please, consult the web page).

It can be observed that the blue color of the sea is considered the background, the yellow of the land, and the dark blue coast line dominates on it, so an erosion decreases the size of the land. It also eliminates small details like letters and lines, whose colors are less abundant, as expected from an erosion. Conversely, the dilation creates a continuous line from the dashed lines and highlights the demarcation lines. This simple method works very well in this image which has only a few different colors and in principle should work also in any image in which the histogram has all the pixels concentrated on a few values, but is not so effective in more complex images with a sparse histogram. With respect to the comparison with the method of Angulo (RGB space, give almost identical results) whose results are shown in Figure 2, the output of the basic morphological operations is almost the same. We could point out that our erosion is more uniform on colour blackground results (blue, yellow, and bright brown), although Angulo’s erosion cleans better the letters. His dilation and corresponding closing highlight a little bit more the demarcations lines and letters on the map. The inconvenience of this method is the need of choosing a reference color. Indeed, choosing white instead of black as the reference inverts the results of erosion and dilation which makes the results counterintuitive but there is no rule to decide which is the best reference color for each image.

5.2. Results Obtained by Smoothing the Histogram of the Image Itself

Ordering obtained from the smoothed histogram of the image itself is tested with a painting from Joan Miró. Figure 3 shows the painting (a) together with its erosion (b), dilation (c), opening (d), and closing (e) with a disk-shaped structuring element of 3 pixel radius using the Euclidean distance in the RGB and color spaces. The white background is the most abundant color and the greenish almost white tonality in the lower left corner contributes to the potential of the white background so erosion eliminates the thin lines, independently on over which variation of the background they run. Note the annulment of Miró’s signature in the lower right corner after an erosion or opening. In this case the use of an artificial (RGB) or perceptual () color space does not have a great influence; the only noticeable differences are the more clear definition of the thickened lines in the dilation and the sharper separation between color regions when using RGB.

In general, RGB should be preferred in images of artificial things with a small number of colors.

Comparison with the method of Angulo can be seen in Figure 4. Again in this case results are visually very similar, and as it has been said before our erosion is more uniform in color than Angulo’s one, as it can be appreciated at the contour of the Chanteuse which is more delimited by our method, specially compared with Angulo’s space for which yellowish and greenish spots are not taken out at all. Nevertheless, in the dilation the black contour of the Chanteuse is better enlarged by Angulo’s method due to the proper choice of black as the reference color.

The next example (Figure 5) shows the application of the same method to a natural image, the dragon sculpture in the Güell Park (Barcelona), using the Euclidean distance. In this case the gray tonalities play the role of background and in general an erosion extends this background over colored patches on the dragon’s body. The opening makes these patches larger and more uniform than in the original images. This effect is more clearly visible in the RGB. Nevertheless, the behavior is the opposite for the totally natural background with leaves on a dark area. Here the leaves are more uniformly treated in the space, probably because their different green tonalities have influence on each other. The method of Angulo whose application is shown in Figure 6, even giving similar results, shows a less blurred image after an erosion than ours.

5.3. Results Obtained by Smoothing the Joint Histogram of Similar Images

The following example shows the application of morphological operations on images not used to build the cube of potentials, so they can be considered as crossed tests. The images used to build the color histogram have a close resemblance in their general look to the target image. Figure 7 shows the groups of images used to create the cube of potentials for the dragon images.

The crossed test using Euclidean distance is shown in Figure 8 which must be compared with Figure 5. Differences of quality are hard to see but the results appear to be slightly better than in the self image method (as in the former case, only for the RGB space). When using the color patches after an opening appear more uniform and the separations between tesseras of the mosaic better were eliminated. This may be due to the image being more rich in colors and to the influence of many colors coming from different images, which smooths more the result.

A last experiment is done to compare the usual gray morphological operations with those implemented with this approach. In order to do that the same color image has been converted to gray, getting only the intensity band, and the four basic morphological operations have been applied to it. Also, the operations have been applied according to our proposal, using the Euclidean distance and the RGB color space (this was already done in the previous experiment) but the results have then been converted to gray, again keeping only the intensity component. Comparisons between these two procedures are shown in Figure 9 for our method and the method of Angulo.

In general we can say that, for a given size of the structuring element, the effect of the different morphological operations makes the image become coarser with our method than with the usual definitions (the image, either visually or in its binary content is more severely modified). Some details that become different between both methods (only clearly visible in the web page) are the elimination with an erosion of the thinner lines in the handrail behind the dragon, their thickening with a dilation, or the conversion of the eye of the dragon by an opening into a single homogeneous patch.

The general conclusion is that the objectives of the morphological operations (elimination of small structures in the case of erosion and opening, and filling of small holes in the dilation or closing) have been successfully accomplished by using any variant of the presented method, and for both of the analyzed color spaces. Hence, the choice of which of these variants should be used for a particular case depends mainly on the number of different colors in the image, the number of images of similar type available, and the computational cost one is willing to accept. In the case of images with large planar patches and few colors the application of direct ordering based on no smoothed histogram is simple and fast. In other cases, if several images are available and the operations must be applied on a large set (like an image database) the construction of the smoothed joint histogram is preferred, since it takes more time than building the simple image histogram; but the cube of potentials can be used for many images. Finally, if only one complex image is available the smoothing of its own histogram is the only possible method. With respect to the color space, a preliminary conclusion might be that simple images with a moderately high number of colors can work well in RGB so no transformation to or other spaces is needed. The perceptual space should be used only for complex natural images. The influence of the distance used to smooth the histogram appears to be minimal, so in general Euclidean it can be used because of its lower computational cost.

5.4. Morphological Gradient

A possible application of these colour morphology definitions is the calculation of morphological gradient. In gray levels, morphological gradient is normally defined as the difference between dilation and erosion of the same image with the same structuring element. As the colour difference is not defined, we cannot extend this definition in a trivial manner. We suggest that, since the potential calculated for each colour is a measure of its importance, difference between the potentials associated to each colour in the dilated and eroded image can be a reasonable generalisation of the morphological gradient. Figure 10 shows the so-defined morphological gradient of the Miró painting using as the structuring element a disk with radius 3 and 5 pixels in both colour spaces. Figure 11 shows the same results on the dragon image. Since the actual value of the gradient, being obtained from fictitious potentials, has not direct physical counterpart, the gradient images have been normalised, with brighter being the higher values of the gradient.

As it can be seen, borders are reasonably well detected, even with some noise. In this case the space gives better results than the RGB. This is because both, erosion and dilation, tend to create more uniform colour patches in this space which improves the border detection and the uniformity of the inner parts of the shapes when both are subtracted. The two sizes used for the structuring element show two different levels of detail for the borders, as expected, and this opens the possibility of designing filters tuned to several resolution scales, in the Marr-Hildreth style.

6. Conclusion and Further Work

This paper has presented a method to generalise the basic morphological operations to colour images. After stating the requirements that such a generalisation should fulfill and reviewing previous works we conclude that any method based on a canonical ordering of the vector colour spaces suffers from a fundamental drawback: it cannot capture the intuitiveness of the expected results for every possible image. Therefore, we renounce to try out this approach and, on the contrary, we provided a method to construct a concrete colour ordering for each particular image. The basic idea of morphology (detect and analyse objects on a background) provides us the clue to this ordering: the most abundant colour should be considered background, and the importance of the colours is measured inversely to its frequency of appearance. This generates two important inconveniences: colours in images with very spread colour histograms are poorly ordered, so a previous preprocessing must be done for each image. The first problem is addressed by smoothing of the colour histogram using a potential-like kernel function and the second, by precalculating the ordering using a set of images similar to the target one. Results are presented and qualitatively assessed in representative images. We think that the inconveniences of image dependency and expensive computation are made up by the intuitiveness of the results and the possibility of precomputing colour ordering for the type of images at hand.

Further work includes the automatic detection of image types based on the resemblance of its colour histogram. This may help on building a set of classes of images with precomputed potentials and later, on deciding automatically which of these classes should be used for a new image. Also, more experiments could be done using different colour spaces and various smoothing kernels.

Acknowledgment

This work has been supported by Grant DPI2008-06691 from the Spanish Ministry of Economy and Innovation.