Abstract
Harris corner detectors, which depend on strong invariance and a local autocorrelation function, display poor detection performance for infrared (IR) images with low contrast and nonobvious edges. In addition, feature points detected by Harris corner detectors are clustered due to the numerous nonlocal maxima. This paper proposes a modified Harris corner detector that includes two unique steps for processing IR images in order to overcome the aforementioned problems. Image contrast enhancement based on a generalized form of histogram equalization (HE) combined with adjusting the intensity resolution causes false contours on IR images to acquire obvious edges. Adaptive nonmaximal suppression based on eliminating neighboring pixels avoids the clustered features. Preliminary results show that the proposed method can solve the clustering problem and successfully identify the representative feature points of IR breast images.
1. Introduction
Infrared (IR) images with advantages of noninvasion, no risks, and low-cost can reveal the heat distribution on the surface of human body. Cancerous tissues tend to have a higher temperature signature than surrounding normal tissues [1] and infrared image has been widely used and suggested in medical applications as a potential assessment tool for breast tumor detection and for the high-temperature tumor generating heat areas on infrared images [2]. If longitudinal breast IR images in the spatiotemporal domain are aligned well, the chemotherapy response could be evaluated and even tumors might be identified by the change of heat areas [3, 4].
In image registration, it is necessary to assign representative control points to the transformation model for good performances of registration results [5].
However, detection of representative control points (feature points) on IR images is generally very difficult. In some medical modalities such as X-ray mammograms, MR images, and ultrasound images, anatomic features, that is, nipples and boundaries, can be found easily and are available to be used as feature points to build relationships for transformation models [6–8]. For infrared images, no anatomic fiducial markers could be found on body surfaces. Therefore, the best suited feature point detection method for IR image is essential and important.
Many feature point detection approaches are based on edge, corner, curvature, singular values, and wavelet to detect representative points [9]. One of the most intuitive types of feature points is the corner. Corners are image points that show a strong two dimensional intensity change, and are therefore well distinguished from the neighbouring points [10]. Corner detectors have been widely used as feature point detectors because comers correspond to image locations with a high information content, and they can be matched among images reliably [10]. Corner detectors are mainly divided into three categories: Moravec corner detector, Harris corner detector, and Fast corner detector. To be useful for IR image feature point matching, a corner detector should satisfy the following criteria: (1) consistency: detected positions should be insensitive to noise variation and, more importantly, should not move when multiple images are acquired for the same scene; (2) accuracy: corners should be detected as close as possible to the correct position; and (3) effectiveness: all possible useful corners should be detected. Moravec corner detector has several obvious defects: (i) the calculation of strength value is not isotropic because only eight discrete 45-degree angles are considered; (ii) since the window is square and binary, the response function has noises; (iii) the edge response is too simple because the strength value is selected by SSD minimum value. For corner detection, Fast corner detection algorithm, though shorter in computation time, is not so rigorous as Harris corner detection, and lots of useful corners are accordingly lost [11].
Therefore, Harris corner detector [12, 13] is selected in this study to detect feature points for IR images due to the strong invariance to rotation, scale, illumination variation, and image noise. Nevertheless, infrared images of low contrast and unobvious edge are poor in detection performances for Harris corner detector which mainly depends on the edge strength and the image intensity. In addition, feature points detected by Harris corner detector are clustered because of the great amount of contained nonlocal maximum.
In this study, a modified Harris corner detection algorithm is developed to overcome low contrast and unobvious edge issues, to avoid corner clustering, and, finally, to identify the meaningful control points. The most important process of the proposed algorithm is to apply suppression radius and how to determine its length. In this study, an innovation method named adaptive suppression radius selection is proposed not only to solve the corner clustering but also to avoid losing useful corners caused by the overlength suppression radius.
Moreover, this method automates the choosing of suppression radius. Furthermore, feature points detected by the proposed algorithm on hot patterns cloud also improve the accuracy of breast infrared image registration for chemotherapy assessment and early detection [14, 15].
2. Material and Method
2.1. The IR Spectrogram System
The core of the IR spectrogram hardware system is a pair of MIR and LIR cameras with high temperature sensitivity and spatial resolution. The MIR and LIR cameras that we have chosen are made by FLIR Systems, both with 320 × 256 pixel detectors. The temperature sensitivity is better than 0.02°C and the spatial resolution at 1 m is about 0.6 mm.
The blueprint of the IR spectrogram system is sketched in Figure 1. The MIR and LIR cameras are placed on top of a platform and the system has been equipped with seven degrees of freedom. As indicated in Figure 1, each camera is allowed to freely move along the x-direction and rotate on the x-y plane, which accounts for four degrees of freedom.
An image-based calibration system has been built in to ensure that the central optical axes of both cameras are on the same plane as shown in Figure 2. The platform supporting the two cameras is allowed to move along the x- and z-directions, which give the fifth and sixth degrees of freedom. The chair to be seated by a subject is rested on a high-rise base positioned in the y-direction. The location of the chair can be adjusted along the rail on the base, which contributes to the seventh degree of freedom. The IR spectrogram hardware system has been designed to provide an imaging environment with high reproducibility and flexibility to assist the development of the IR spectrogram imaging modality for practical use. The high reproducibility is required to assure that a subject is seated in the same location (relative to the cameras) when this subject is taken IR images at different times. The high flexibility is required to support finding the best camera positions for different subjects.
(a)
(b)
Figure 3(a) shows the actual system constructed according to the blueprint shown in Figure 1. Each subject is seated on the chair and put hands behind head as shown in Figure 3(b). Figure 4 gives a closer look at the platform supporting MIR and LIR cameras. The three dimensional coordinates of the chair and cameras as well as the azimuth angles of both cameras may be recorded for subjects with multiple checkups. The system may be augmented to robotic control by using motors if necessary.
(a)
(b)
2.2. Clinical Test
Clinical tests are carried out under the approval of the Institutional Review Board of National Taiwan University Hospital before taking IR spectrograms. Every subject needs to sign a patient consent form after it has been explained by a doctor or a research assistant. The subject should take off all clothes on the upper body once entering the checkup room and above ten markers are adhered to the upper body around breasts. Finally, all images to be analyzed will be provided by the assistant with anonymous image ID number.
2.3. Algorithm
The feature point detection in this study applies the Harris corner detector, the basic idea of which is the local autocorrelation function of a signal that measures the local changes of the signal with patches shifted by a small amount in different directions. The flow chart of the proposed algorithm is shown as Figure 5.
The weighted Sum of Squared Differences, denoted by , is used to calculate the point magnitude of grayscale-change. Given a shift and a point , the autocorrelation function is defined as where is the image and are the points in the window (Gaussian weighting factor) centered on . The shifted image is approximated by a Taylor expansion truncated to the first order, which terms Rewriting (1) leads to the approximate equation which could be written in the matrix form where is the structure tensor
In the Harris corner detector, the position of the corner points is determined by the two characteristically higher values for structure tensor . The cost function is , where is usually between 0.04 and 0.06. A corner is defined as a cost function value exceeding the threshold.
However, the low contrast of the infrared images makes it difficult for the Harris corner detector to detect corners, due to its reliance on edge strength and image intensity. In addition, feature points detected by the Harris corner detector tend to be clustered due to the large number contained within the nonlocal maximum.
2.4. Image Contrast Enhancement and Adjustment of Intensity Resolution: To Overcome Low Contrast and Lack of Obvious Edges
This section addresses the difficulties involved with infrared light by enhancing contrast via histogram equalization (HE) and adjusting the intensity resolution to obtain false contours.
We initially assume that the image grayscale has a normalized form to in which the variable represents a value from the original grayscale prior to image enhancement.
Equation (6) transforms variableto a new grayscale:
The transformation function must be single valued and monotonically increasing in , to ensure that the mapped grayscale values remain within the acceptable range from black to white. Let be the discrete grayscale image and be the number of occurrences of grayscale level , such that the probability of a pixel of level occurring in the image is determined as follows: where is the total number of gray levels in the image and is the total number of pixels in the image.
The general formula for histogram equalization is as follows: where and gives the number of pixels in the image.
Following contrast enhancement, false contours were obtained by reducing the intensity resolution. This study used false contours with intensity resolutions of 16 and 2. More details can be obtained using an intensity resolution of 16, but an intensity resolution of 2 provides additional edge information; therefore, we integrated the two.
This study used these two intensity resolutions to coordinate the Harris corner detector for corner detection and the selection of feature points in IR images.
After increasing the contrast of the original image, as shown in Figure 6(b), the intensity resolution is adjusted to enhance the edges for corner detection, as shown in Figure 7.
(a)
(b)
(a)
(b)
2.5. Adaptive Nonmaximal Suppression: To Avoid Corner Clustering
The size of breast tumors may change during the period of chemotherapy treatment; therefore, this study sought to register the entire breast area without producing distortions in the rendering of the tumor. This requires an even distribution of feature points in all areas except the specific location of the tumor. This study modified Harris corner detection to avoid the problem of clustering. Figure 8 outlines the steps involved in the proposed method.
3. Adaptive Radius Suppression
In order to prevent clustering, this study sought to ensure that the relative distances between points were close to equal by increasing the radius. However, oversuppression of the radius would eliminate many useful corners resulting in poor registration results. Figure 9 illustrates the distribution of corners in five cases (C43, C44, C51, C52, and C55) by varying the radius suppression. These results indicate that oversuppression resulted in the detection of only a few corners and undersuppression resulted in corner clustering.
Figures 9(a), 9(d), 9(g), 9(j), and 9(m) present corner distribution with a suppression radius of 1, resulting in pronounced corner clustering. Figures 9(b), 9(e), 9(h), 9(k), and 9(n) present the results obtained when radius suppression was selected adaptively, where corner distribution is evenly distributed and a sufficient number of corners are retained. Figures 9(c), 9(f), 9(i), 9(l), and 9(o) present the results obtained when the radius undergoes oversuppression resulting in the elimination of many otherwise valuable corners.
Figure 10 outlines the steps involved in the proposed method for the adjustment and selection of radius suppression.
Table 1 presents a matrix of eliminated corners, , based on the number of corners eliminated when the radius is increased by 1. Figure 11 presents the curve formed by applying the elements of . The length corresponding to the highest peak is used to identify the primary suppression radius. The curve representing eliminated corners in an IR image is a convex function. The maximum number of eliminated corners indicates the points at which the corresponding radius r is the relative distance between the majority of points.
4. Verification
The corner distribution of a normal IR image produced by the proposed algorithm indicates three areas: (1) clustered areas where the distribution is dense but the number of corners is below that found in areas of homogeneous distribution; (2) homogeneous distribution areas where the distribution of a large number of corners is uniform; (3) discrete areas where distribution is discrete and the number of corners is below that found in areas of homogeneous distribution. To verify the validity of the method, we created the test image in Figure 12 with clustered, homogeneous, and discrete areas from left to right. We obtained values from a random number generator instead of using the local maximum value in the conventional Harris corner detector, to provide radius suppression from 1 to 12 pixels, as shown in Figure 13. Figure 14 lists the number of eliminated corners and changes in their distribution.
Clearly, increasing the radius suppression to the length associated with the first peak reduced corner clustering; however, this only reduces few corner clusterings. Increasing radius suppression to the length associated with the highest peak resulted in uniform distribution. In this case, the corner distribution distance is greater than the highest peak r + 1; therefore, the suppression radius cannot be further increased without corners being lost, and whereby no enhancement in uniformity would be gained.
The last step involves evaluating the quality of corners. Because there is only one point in suppression radius’ cover area, evaluating corner clustering avoidance relies on efficiently increasing the number of corners. If the number of corners detected by adaptive suppression radius selection is less than one third of the original number, reducing the suppression radius to can ensure that a sufficient number of corners are available for matching and registration.
As shown in Figure 16, the curve obtained from the proposed algorithm is smoother than the original in Figure 15, indicating that the proposed algorithm detected fewer corners at the same radius. This also means that the proposed algorithm is able to overcome the problem of corner clustering.
5. Results and Discussion
Figure 17(a) shows the corners detected using the Harris corner algorithm in the original image and in the enhanced image, respectively. Clearly, a greater number of corners are visible in Figure 17(b).
(a)
(b)
Figure 18 presents the results of corner detection: (a) intensity resolution of 16; (b) intensity resolution of 2. The two intensity resolutions detect corners in different regions of IR images with complementary effects.
(a)
(b)
This study eliminated the problem of corner clustering in the Harris method for the detection of features, as shown in Figure 19.
(a)
(b)
By combining the corners in Figures 19(a) and 19(b), we can obtain all of the useful feature points in the IR image, as shown in Figure 20. Feature points are located in the crucial hot pattern positions and do not affect the areas of the tumor, as shown in Figure 17.
6. Conclusion
This study modified Harris corner detection for infrared images in order to reduce corner clustering, increase contrast, and make the edges easier to detect. The adaptive radius suppression technique reduces corner clustering, avoids the loss of useful corners due to oversuppression, and automates the selection of suppression radius. Moreover, corners detected by the proposed algorithm on hot patterns are representative and improve the accuracy of infrared image registration for chemotherapy assessment and early detection.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
Research supported by National Science Council of Taiwan under NSC Grants: NSC 102–2218-E-239 −004 -MY2.