Abstract
Computer vision algorithms that use color information require color constant images to operate correctly. Color constancy of the images is usually achieved in two steps: first the illuminant is detected and then image is transformed with the chromatic adaptation transform (CAT). Existing CAT methods use a single transformation matrix for all the colors of the input image. The method proposed in this paper requires multiple corresponding color pairs between source and target illuminants given by patches of the Macbeth color checker. It uses Delaunay triangulation to divide the color gamut of the input image into small triangles. Each color of the input image is associated with the triangle containing the color point and transformed with a full linear model associated with the triangle. Full linear model is used because diagonal models are known to be inaccurate if channel color matching functions do not have narrow peaks. Objective evaluation showed that the proposed method outperforms existing CAT methods by more than 21%; that is, it performs statistically significantly better than other existing methods.
1. Introduction
One of the pillars of color science is the illuminant of the image being treated. Many real-life applications in the field of computer vision require images that are invariant to the illuminant changes. In [1] authors discuss a problem of the fluorescent lamp spectral distribution change over time in their computer vision system for classifying marble plates. Face detectors [2, 3] and face extractors [4] use predefined skin colors to segment the image. These colors accurately present skin tones only on images with standard illumination. Color- and texture-based image search [5] also requires images to be described with illuminant invariant descriptors.
The mechanism of human vision system that takes care of the illuminant invariance is called color constancy [6]. In digital world it is usually modelled as a two-step process consisting of illuminant estimation and image transformation [7]. This paper discusses the second step of the process, that is, image transformation.
Image transformations are done with chromatic adaptation transforms (CATs). The conversion from one illuminant to a different one has mostly been handled by using single diagonal von Kries-like transformation [8] for all the colors in the gamut of the input image. The main reason for using a single diagonal model is the fact that usually only one corresponding color pair under two different illuminants is known (i.e., source and target illuminants); thus there is not enough information to construct more complex models. But if there are many known corresponding color pairs, there is no need to limit ourselves to one diagonal transformation; instead we can use a higher number of more complex transformations.
Diagonal von Kries like transformations scale color channels independently. Whether or not human color constancy operates in the same way is beyond the scope of this paper. But it has been shown [9, 10] that diagonal von Kries like transformations do not accurately model chromatic adaptation if color channels are not independent of each other; that is, channel color matching functions have wide peaks. Sensor sharpening [11] has been used to derive new color matching functions with narrow peaks but some correlation between different color channels still remains even after the sharpening process. Full linear transformations can be used to account for the correlation between different color channels.
The method proposed in this paper uses Delaunay triangulation and Macbeth color checker to divide the color gamut of the input image and it constructs one full linear transformation for each triangle of the divided color gamut; that is, each color in the input image is transformed with the matrix of the triangle that contains the color. It is meant to be used primarily as a preprocessing step of other computer vision algorithms operating with color information.
Similar approaches have been used to transform colors from camera color space to device-independent color space. It should be noted that our approach addresses the chromatic adaptation and assumes that colors have already been transformed from camera to device-independent color space. Granger [12] divided color space based on the hue values of reference color points and calculated transformation matrix for each one of the subspaces. Andersen and Hardeberg [13] used patches of the Macbeth color checker to divide color space and associated each subspace with linear transformation that preserves neutral colors and hue planes. Both of the approaches use two-dimensional chromaticity space to find subspace that contains specific color, but actual transformation is done using all three channels of the original color space. Our approach differs from that in [12, 13] by defining a new way of dividing color space using Delaunay triangulation and using chromaticity space both to find subspace and transform color points.
The rest of the paper is organised as follows. Section 2 describes the problem and related work. Section 3 provides the description of the new method. Section 4 gives experimental evaluation of the new method. Finally, conclusions are drawn in Section 5.
2. Problem and Related Work
Chromatic adaptation is the ability of the human visual system to adjust to illumination changes and preserve color appearance of the objects [14]. It allows us to see stable colors of the objects illuminated by a wide range of different illuminations. Chromatic adaptation transforms (CATs) are methods used in digital imaging and color science to model the described mechanism of the human visual system. They provide a means to transform color values under a source illumination into color values under a target illumination.
A standard model to compute transformation from one illumination to another one is diagonal von Kries like adaptation model [8]. If denotes color value under source illumination, then model states that we can model the same color value under target illumination as where , , and represent scaling coefficients for color channels. These scaling coefficients are most often the ratios of target illumination and source illumination ; that is, , , and . However, different CATs differ in the color space in which this scaling takes place.
Obvious choice is the color space in which image is initially described, such as sRGB color space. This process is simple as no additional transformations of the color spaces are required. Other commonly used color spaces are derived as linear transformations of the space [15]. The process of deriving these linear transformations of the space is called sensor sharpening [11]. Color matching functions of the derived color spaces tend to have sharper, narrower peaks, thus they support better the von Kries like models. The basic model for transforming color values in derived color spaces is as follows.(1) value is transformed into value.(2) value is transformed using (3) value is transformed back to value. Some commonly used transformations of the color space, , are as follows. (i) : is the identity matrix in (3). This method's performance is poor [16]: (ii) Bradford: a commonly used transformation derived by comparing 58 samples of color wool under illuminants D65 and A [17]. At the beginning this transformation was nonlinear in the blue channel, but this nonlinearity was eliminated later. Consider (iii) Sharp: a widely used newer transformation derived from the Bradford transform [18]. It has more narrow peaks of color basis functions than the Bradford color space; thus it supports better von Kries like model. Consider (iv) CMCCAT2000: simplified CMCCAT2000 [19] is the successor of the complex CMCCAT97 transformation [20], which was also derived from the Bradford transformation. CMCCAT2000 also allows modelling of the partial adaptation of the observer, but partial adaptation is not considered in this paper because we assume a fully adapted observer. Consider
All of the described transformations use the diagonal von Kries like model to model illuminant change. But it has been shown [9, 10] that diagonal model is not accurate if color channels are correlated. Even after color channels are transformed using sensor sharpening there is still some correlation left. We can model the correlation using full linear model instead of the diagonal model. The reason why usually the full linear model is not used is that there is not enough information available to determine the coefficients, in particular, if we only know the values of the illuminants. But if we have more color correspondence pairs between two illuminations, there is no need to limit our methods to the diagonal model. Instead we can use full linear model that also captures correlations between color channels.
3. Proposed Method
The algorithm proposed in this paper is based on the assumption that color points whose chromaticity values are close together in the color gamut of the source illumination should warp smoothly to a new color gamut of the target illumination. In order to change the illuminant of any color point, the color gamut is divided into smaller regions according to some reference points, and the transformation to be applied to each region is computed. That is, the objective is to obtain the transformation matrix to be applied to each region in the source color gamut. Our method is suitable for controlled applicative environment, because it assumes that both source and target illuminations are known and that the Macbeth color checker is present in the input image, or color values of the patches of the Macbeth color checker captured under the illuminant of the input image are known.
The following subsections explain our transformation between the source and the target color gamut, outline the Delaunay triangulation used to divide the color gamut into smaller regions, and finally describe the proposed CAT method to change the illuminant.
3.1. Calculation of the Transformation Matrices between Different Color Gamuts
The function of transformation to be applied for changing the illuminant can be obtained if some reference points are known in the source color gamut and their transformations in the target color gamut are known as well. We describe these reference points in chromatic rg space. The reason why we are using the chromatic rg model instead of the native RGB model is that we want similar colors, ignoring intensity values, to be transformed with the same transformation matrix.
The following equations system in (7) is the mathematical relation between one chromatic point in the source color gamut and the target color gamut . This relation can be linear, quadratic, cubic, and so forth, depending on the value of . The quantities and are unknown factors that can be solved if several points in the source color gamut and their transformation in the target color gamut are known. Consider
Any point within the source color gamut can be transformed into the corresponding point of the target color gamut by using the relation in (7).
If a linear relation is considered, then the system in (7) becomes or in matrix form where
In order to calculate the unknown factors in : and for as there are six unknown factors and two equations, it is necessary to know at least three points in the source color gamut and their transformations in the target color gamut. If we had chosen a higher order relation more correspondence points would have to be known.
The known three points in the source color gamut for , describe a triangle. They can be arranged in matrix form as
And the known three points of the transformed triangle in the target color gamut , for , can be arranged in matrix form as
The equations system to be solved in order to compute the unknown factors in is
And the solution to this system is where can be computed by means of a singular values decomposition (SVD) [21].
Transformation matrix differs from diagonal models used by other CAT methods by also capturing the relations between separate color channels; thus it is more accurate if color channels are correlated.
Once the transformation matrix is known, it is possible to transform any point within the triangle defined by the known reference three points in the source color gamut to the target color gamut by using (9). The smaller the triangle in the source color gamut, the more accurate the transformation will be for the points inside the triangle. The transformation is completely accurate when transforming vertices of the triangle, because they have direct mappings from source to target illuminant. So if we reduce the size of the triangle, color points that lie inside it become more similar to the vertices of the triangle which have direct mappings defined. This is the reason why it is desirable to divide the color gamut into the smaller possible triangles according to the reference points known in both color gamuts. The method explained in the following subsection is used to obtain the biggest number of nonoverlapping triangles given a number of reference points.
3.2. Delaunay Triangulation
Delaunay triangulation [22] is a well-known method in the field of mesh generation due to its optimality properties. In the plane or the 2D case, the Delaunay triangulation of a set of points , for in , is a collection of triangles , such that no point in is inside the circumcircle of any triangle in .
For a set of points, the number of triangles produced by triangulation depends on the spatial distribution of the points. If is a number of points forming convex hull of the points, then the number of triangles is at most
Degeneration of can occur, if all the points lie on the same line. Then there are no triangles defined. But considering the domain of color transformation this is unlikely to happen.
The Delaunay triangulation will be used to divide the color gamut into smaller regions (triangles) according to the reference points known in both color gamuts. We should mention that the resulting triangulation is not unique when four or more points lie on the same circle. Further research will be needed to study the effects of different possible triangulations of the same color points.
3.3. Illuminant Change Method
Once the formula for the transformation matrix for a number of reference points is known and the way to divide the color gamut into smaller triangles has been established, then, in order to perform the illuminant change, the following steps are to be carried out.(1)The Macbeth color checker (Figure 1), present in the input image, is used as the reference to achieve color constancy. Delaunay triangulation is done to divide the original color gamut by using the mean values of the 24 patches of the Macbeth color checker. Figure 2 shows triangulations of the color gamut for two different illuminants.(2)Once the triangles are defined, the transformation matrix of each triangle in the source color gamut is computed using (16). Thus, there are as many matrices as triangles.(3)For each point of the source color gamut that needs to be transformed, the transformation in the target color gamut is obtained with (9), that is, by applying the transformation matrix of the triangle that contains the point. First, RGB color point is converted to rg point . Then the triangle containing point is found and the point is transformed using the transformation matrix of the triangle containing it, resulting in a new rg point . The new RGB point is obtained using preserving the sum of channel intensities before and after the transformation.
As far as the points in the source color gamut are inside one of the triangles obtained by Delaunay triangulation using the patches of the Macbeth color checker, the resulting transformation using the matrices is valid. Nevertheless, the triangles defined by the patches of the Macbeth color checker do not cover the whole rg chromaticity space. Hence, problems appear when transforming a point in the source color gamut that is outside any triangle, as the illuminant is not corrected properly for these points.
In order to handle these points outside the triangles defined by the patches of the Macbeth color checker, we consider the vertices of the whole rg chromaticity space as reference points together with the patches of the Macbeth color checker. The vertices of the rg chromaticity space in are , , and . Note that point is not the vertex of the rg chromaticity space. By the definition of rg space, sum of both components cannot be greater than one [23]. The transformed points of these vertices for any illuminant are the same points, as the limits of the rg space are fixed and if an overflow occurs the obtained point is saturated to the limit.
The proposed method has one restriction. The triangles defined by the values of the Macbeth color checker under a source illuminant may cross or fold over each other when transformed to a target illuminant. Likewise the transformation of the arbitrary color point within certain triangle under the source illuminant does not guarantee that transformed color point will be contained in the corresponding triangle under the target illuminant. Because of these properties the inverse transformation cannot be defined with our proposed method. However exact inverse transformation is rarely needed in the applicative environment described at the beginning of the paper.
4. Experimental Evaluation
To objectively evaluate the performance of our method, we used color-checker image database [24]. It contains 568 images captured under different artificial and natural illuminations. All the images in the database contain a Macbeth color checker, which is an assumption our method implies. The coordinates of the patches of a Macbeth color checker are available on the website of the image database [25], from which we also obtained illuminant estimates used by other CAT methods. To speed up evaluation, all transformations were made on the downscaled versions of images also available on the website of the image database.
All the images in the database are described in sRGB color space with gamma correction applied. CAT methods assume that linear images are given, so inverse gamma correction was applied to produce linear images [26]: where represents the linearized sRGB value of the color channel .
To evaluate the performance of our method, we transformed the color patches of the Macbeth color checker on each of the 568 images and compared the transformed values with the values of the patches of the Macbeth color checker captured under standard D65 illuminant [27]. To ensure that our method is not in a privileged position compared to other CAT methods we took two precautions as follows.(i)The color value of the patch currently being transformed cannot be used when triangulating the source color gamut and calculating transformation matrices. Otherwise it is the part of the transformation matrix.(ii)Chromaticity values of the patches in the last row of the Macbeth color checker are very similar. That is the reason why we ignore all the patches of the Macbeth color checker in the last row except the third patch from the left. One could also choose any other patch from the last row.
Performance of our method is evaluated using the following steps. (1) Color values of the patches of the Macbeth color checker in the input image are calculated.(2) All the color patches from the last row of the Macbeth color checker except the third one from the left are discarded.(3) To each of the remaining 19 color patches we apply the following:(a) From the set of color values we temporarily remove the value of the patch currently being transformed.(b) Delaunay triangulation is applied.(c) We find the triangle containing the color value of the patch currently being transformed.(d) We use the transformation matrix of the triangle to transform the patch value.(e)Difference between the transformed value and the reference value of the patch under standard D65 illumination is computed.(4) The error of the single image is equal to the mean difference of all the transformed color patches.
Performance of other CAT methods (sRGB, , Bradford, Sharp, and CMCCAT2000) is evaluated similarly as follows.(1)Color values of the patches of the Macbeth color checker in the input image are calculated.(2)All the color patches from the last row of the Macbeth color checker except the third one from the left are discarded.(3)Color values of the 19 remaining patches are transformed using the CAT method. Differences between the transformed value and the reference value of the patch under standard D65 illumination are computed.(4)The error of the single image is equal to the mean difference of all the transformed color patches.
To compute the difference between the values of transformed patches and reference patches, perceptual euclidian distance (PED) is used: where represents normalised transformed values, represents normalised reference values, and , , and represent weight coefficients. It was shown that PED with weight coefficients , , and finds its roots in human vision and correlates significantly higher than any other distance measures [28].
Mean PED errors of different methods are shown in Table 1. Performance of the evaluated methods is similar. The only method that stands out is our proposed method (DT-based) that has significantly lower mean PED error than the other methods. The second best methods (Sharp and CMCCAT2000) have more than 21% greater mean PED error. This is an encouraging result for the use of the proposed method.
Figure 3 shows box plots of PED errors for different CAT methods. Box plot is a tool for visualisation of the data distribution [7] or in our case a tool for the visualization of the distribution of the PED errors. It visualizes five statistics: minimal error , lower quartile , median error , upper quartile , and limited maximal error , which is not greater than . Errors greater than are shown as outliers.
Although the evaluated CAT methods have similar box plots, it can be clearly seen that our proposed method has lower , , , and ( for all the methods). This tells us that most PED errors of our proposed method are lower than corresponding PED errors produced by other CAT methods.
To verify that our proposed method performs better than other existing methods, we conducted statistical test as suggested in [7]. Mann-Whitney U test [29] is a nonparametric statistical test that does not imply normal distribution of underlying values. It is used to show whether median values of two samples are statistically significantly different or not. Two values are significantly different if, given the confidential level, we can conclude that the observation is not the result of a random process [30].
We compared median PED errors of each pair of evaluated methods and checked if two values are significantly different. Results are presented in Table 2. Median PED errors of methods sRGB, , Bradford, Sharp, and CMCCAT2000 are not significantly different, except for the significantly lower error of the Sharp transform compared to the transform. The proposed method has significantly lower median PED error compared to all of the other methods. We conclude that under the experimental conditions presented in this paper our method significantly outperforms other existing methods.
Surprisingly the sRGB method performs similarly to the methods using sensor sharpening. Some authors [11, 15] state that methods based on RGB color model do not perform as well as sensor sharpened methods. We showed differently.
Figure 4, showing correlation between the number of triangles produced by triangulation and mean PED error, confirms that the number of triangles has significant influence on the performance of the proposed method. A higher number of triangles imply lower triangle areas in which the transformation matrix more accurately transforms part of the original color gamut. A higher number of known corresponding points between the source and the target color gamut results in more accurate transformation matrices and better method performance.
Figure 5 shows random image from the color-checker image database transformed with conventional method and our proposed method. Original image was shot under artificial light and has strong color cast. Both transformations greatly reduce this color cast when transforming to D65 illuminant. By closer examining image transformed with method (zoom in on the image) you can still see yellow color cast on the ground. On the image transformed with our method this color cast is completely removed.
5. Conclusion
This paper tackles the problem of changing the illuminant of an image and proposes a new method to increase the accuracy of the transform. The proposed method uses Delaunay triangulation and linear transformations in order to transform the input image. The only assumption it relies on is that the color values of the patches of the Macbeth color checker captured under the same illuminant as the input image must be known; that is, the Macbeth color checker must be captured under the same illuminant but not necessarily in the same image.
Objective comparison shows that our method performs significantly better than other state-of-the-art methods.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work was supported by the Ministry of Science of Republic of Slovenia, Program Computer Vision.