Abstract
Infrared thermography has been widely used in various domains to measure the temperature distributions of objects and surfaces. The methodology can be further extended to 3D applications if the spatial information of the temperature distribution is available. This paper proposes a 3D infrared imaging approach based on silhouette volume intersection to reconstruct volumetric temperature data of enclosed objects. 3D IR images are taken from various angles and integrated with 2D RGB images to effectively reconstruct a 3D model of the object's temperature distributions. Various automatic thresholding methods are also compared and evaluated by reprojection scoring to systematically assess the effectiveness and accuracy of the different approaches. Experiment results have demonstrated the ability of the system to provide an estimate to the 3D location of an internal heat source from images taken externally.
1. Introduction
Technology improvements in infrared imaging have facilitated the use of thermal imaging in an increasing number of fields. Most methods only employ 2D thermal images since these are easily obtainable. However, although the thermal images scans can provide the heat distributions as 2D projections, they cannot provide depth or 3D information of the heat source. For example, it is possible to deduce from a 2D thermal image that a heat source exists; however, the heat source’s distance from the scanner cannot be readily deduced from one single image.
The utilization of thermal imaging has been useful in electrical systems, particularly with respect to monitoring or warning applications, where there is often an increase in temperature before catastrophic breakdowns. While temperature increase may not be easily detectable by visuals systems utilizing visible light, such as the human visual system or cameras, such increase in temperature can be observed using infrared thermography apparatus. However, infrared thermography usually provides 2D images for temperature measurement and from the images alone it may be difficult to determine the distance or the relative position of the temperature increase, which may be the location of the failed components.
We propose a method to determining the 3D location of a heat source via multiple 2D infrared thermography (IRT) images acquired using a thermal imager. The proposed method is useful in pinpointing the location of a failed component within closed electrical systems and preventing possible catastrophic breakdowns via early detection. In our work, 2D IRT images are taken around a closed electrical system which encloses a heat source. From the heat distribution in the IRT images and the locations where the images are acquired, we use volume intersection method to reconstruct a 3D model of the system, integrated along with the observed heat distributions. Based on the reconstructed 3D model and the projected heat distribution map, we can determine the location of the most intense region and thus deducing the possible faulty component or reason for failure. In this manner, early and quick detection is made possible solely by external observations without having to take the system apart. The proposed method can be effectively applied to the examination of various power related systems to improve the efficiency in failure detection and the safety of the overall system.
The rest of this paper is organized as follows. Section 2 provides brief background on related work and the selected methods and Section 3 describes the proposed approach. Section 4 provides the experiment results and discussions, and finally, the conclusions are given in Section 5.
2. Background
Infrared thermal images are acquired by thermographic cameras which detect radiation in the infrared range of the electromagnetic spectrum and produce images that visualized the intensity of the radiation. Since all objects above the absolute zero emit infrared radiations, thermography is quite useful in acquiring informative images of the environment with or without visible illumination. Recent works on the acquisition of 3D surface temperature distribution have included structured lighting as a method to acquire 3D surface information [1, 2]. However, the approaches require additional hardware, such as a light source projector, a laser range finder, or other sensors to be used in conjunction to the thermal cameras, thus increasing the complexity of the hardware setup and reconstruction process, as well as the cost in the required computation time. For example, in [3], terrestrial laser scanning, close range photogrammetry, and thermal imagery were used in conjunction to record the 3D data for world heritage monuments. Also, in [4], the authors proposed an approach to map terrestrial and airborne infrared images onto existing building models. On the whole, the integration of 3D and infrared thermal data is still a topic with significant research potentials.
The thermographic device used in this work is equipped with a colour camera which can be used to capture the shapes and colours of objects within the scene. In the proposed method, we intend to perform 3D reconstruction of the object using the acquired colour images. After a preliminary investigation and survey, we eliminated methods which require sophisticated setup and equipment, as well as methods which are sensitive to environmental parameters, and decided to use the shape from silhouette method, also known as shape from contours, for this application. Shape from silhouette was first proposed in the 70s [6], the method requires input images of the object taken from different view angles, and the contours of the object are then extracted from each image and used to “carve out” the 3D shape of the object by intersection with a virtual cube. The method is robust in the sense that a 3D model of the object can be reconstructed as longs as the contours of the object and the view angles of the images are obtainable. The accuracy of the reconstructed 3D models can be improved by providing more details of the extracted contours, improving the accuracy of the estimated view angle, and most importantly acquiring more images of the object from different view angles, covering as much of the object as possible. The shape from contours method does not require sophisticated setup and equipment and can be readily applied to our purposes. In our application, the thermal imager takes regular 2D colour images and the thermal images of the object simultaneously, making image acquisition a one-step procedure. The rest of the processing can then be performed offline.
3. Proposed Approach
We propose a method to determining the 3D location of a heat source via multiple 2D infrared thermography images acquired using a thermal imager as shown in Figure 1, with the specifications of the device given in Table 1 [5].
The proposed method is useful in pinpointing the location of a failed component within a closed electrical systems and preventing possible catastrophic breakdowns via early detection. In our work, 2D IRT images are taken around a closed electrical system which encloses a heat source.
In this paper, we constructed several simulated environments to test the proposed method. Figure 2 shows the concept of the system for our experiment and the steps in the proposed method. A thermal imager is used to capture thermal images of a covered metallic container, which contains a heating element to provide a heat source for detection. The container and the heater are controlled by a control circuit. 2D IRT images captured by the thermal imager from different directions are analyzed and used to reconstruct a 3D model of the object. From the reconstructed 3D model, we can then determine the location of the heat source in 3D space by projecting the thermal distributions from the images. During the entire experiment, the heat source is not visible from outside the container; as a result, the location of the heat source can only be determined by thermal image projections.
(a)
(b)
Although the heat source is not visible from outside the metallic container, by heat conduction, the heat is able to transfer to the surface of the container and radiated outwards towards the sensors. The thermal imager has two sensors; one for the reception of visible light and the other sensor is for the reception of infrared radiation. The thermal imager is able to pick up the radiation and measure the temperature. The other sensor is able to capture colour images of the object. Both the thermal and colour images of the object are processed to remove excess components and to enhance the more significant information. We propose the following steps for processing the acquired images and determining the 3D location of the heat source: automatic thresholding, camera calibration, volume intersection by silhouette, temperature mapping, and reprojection scoring.
Automatic segmentation is used to remove noisy signals from the thermal images. In the thresholding techniques evaluated in [7], Otsu’s method [8] and entropy method [9] have shown better performances. Therefore, in our work, these two methods are selected to perform automatic thresholding of the thermal images.
Camera calibration is used to determine the camera’s locations when the images are acquired [10]. The parameters obtained from calibration are used along with volume intersection by silhouette [11] to obtain the 3D model of the object. The temperature distributions are mapped onto the 3D model also based on the parameters obtained from calibration. Finally, the 3D model is examined to determine the location of the heat source.
3.1. Automatic Segmentation
Reprojection is a common method for evaluation in 3D reconstruction; it is used to determine the accuracy of the reconstructed 3D model. There are several issues for reprojection evaluation [12]; these include having a priori knowledge of the projected depth and the use of iterative algorithms and there is no guarantee of convergence in all cases. This paper adjusts the uses reprojection evaluation and uses it to determine reprojection scoring. Figure 3 provides an illustration of reprojection scoring, where the projected image is segmented into foreground and background via thresholding. There are two steps: the first step is the projection transformation between the different coordinate systems and the second step is to determine whether the reprojected point successfully intersects the foreground image for score calculation.
The proposed method uses automatic thresholding to perform segmentation in the thermal images. In the thermal images, regions with higher temperatures will be shown in lighter colours, as shown in Figure 4, where the vertical scale on the right hand side indicates the temperature corresponding to the gradient of colours. As can be seen from Figure 4, the highest temperature in the image is around 146.6°F, indicated in yellow, and the cooler background regions have temperature of 88.3°F, shown in dark blue.
3.1.1. Segmentation Using Otsu’s Method
There are many different approaches to image segmentation; they can roughly be categorized into threshold-based, region-based, and motion-based [13]. Among these approaches, threshold-based segmentation is better suited to our purposed method. In the thresholding techniques evaluated in [7], Otsu’s method [8] and entropy method [9] have shown better performances. Therefore, in our work, these two methods are selected to perform automatic segmentation of the thermal images.
Otsu’s method was proposed in 1979 and assumes that the image is composed of two classes (e.g., a foreground and a background). The optimum threshold for segmentation is calculated by minimizing the intraclass variance [8]. In Otsu’s method, the intensity variance of the image is represented by as inwhere is the intensity variance, is the with-in class variance, and is the between-class variance. The variance needs to be minimized with respect to the weighted sum of the variances of the two classes as shown inwhere and are the variances and and are the weights of the two classes, respectively. The two classes are defined based on a threshold, , such that pixels with values smaller than are considered to be in class 1, and values equal to or larger than are considered to be in class 2.
On the other hand, minimizing implies maximizing , which is given bywhere is the overall mean intensity and and are the mean intensities for the two classes. These parameters can be optimized by iteratively calculating the class means and variances for a given image to determine the best threshold for segmentation.
3.1.2. Segmentation by Entropy
Entropy has often been used as a measurement across multiple disciplines such as thermodynamics, cosmology, and information theory. Suppose, for an image with intensity values of 0 to 255, the probability of intensity occurring in an image is given by ; the entropy measures for the two classes are given by where represents the threshold used to segment the two classes as described in Section 3.1 and and are the entropies for the two classes. The goal is to iteratively adjust the threshold such that the entropies can be maximized.
3.2. Reprojection Measure
In 3D reconstruction, reprojection error is a common method for evaluation, used in determining the accuracy of the reconstructed model. The reprojection process has been described with respect to Figure 3 in the previous section. In this section, we present the calculation for the scoring.
After the segmentation process, the thermal image is separated into foreground and background. The foreground is used to reconstruct the thermal distribution on the 3D model of the object from multiple viewing angles. After the reconstruction has been completed, surface points on the 3D model are reprojected back into the input image to determine the accuracy of reconstruction. The numbers of pixels that are reprojected correctly into the foreground regions of the input images are counted as the reprojection score. In (5), is the intersection between the reprojection of the surface voxels in the 3D model, , and the foreground of the input image, . Consider
The overall score is the ratio between and . If the reprojections from the surface voxels in the 3D model are correct and match every foreground pixels in the image, a score of 100% would be obtained.
4. Experiment Results
We used three different setups with heat sources to test our approach: a light bulb, a computer case with an internal heat source, and a metal container with an internal heat source. Both the computer case and the metal container’s heat sources are not visible from the outside.
Therefore, we have to rely entirely on the thermal images to determine the locations of the heat sources within the containers. Image of the setup and the reconstructed 3D models are shown in Figures 5 to 7.
In Figure 5, the top left subfigure shows the heat distribution within the 3D model. From the thermal mapped 3D model, the location of the highest temperature can be estimated. The bottom left subfigure shows an extracted subregion of the 3D model to observe the heat distribution more closely without the irrelevant voxels. The right subfigure shows the 3D model being projected onto the 2D image. From the projected image, it can be observed that the highest temperature occurs near the top of the light bulb.
The reconstructed images shown in Figure 6 were not as successful. The reason might be that the computer case is too thick for the internal heat source to be detected from the outside by the thermal imager. Nevertheless, from the top left subfigure which shows the reconstructed 3D model with the thermal mappings, we can still roughly estimate the location of the highest temperature within the case.
In Figure 7, the top left subfigure has been trimmed to show the location of the heat source, since the container is rather large and the heat source is not easily visible without removal of the external voxels. From the 3D models, we can once again estimate the location of the highest temperature within the metal container, even if the heat source is not visible from outside the container.
Tables 2 to 4 show the reprojections measure when automatic segmentations are performed using entropy and Otsu’s methods. The first column of the table provides the index for the different number of views. During the experiments, the objects are placed on a turntable and rotated as images are taken. A total of 12 images are taken for 360° of rotation, with 30° between the acquisitions of successive images. The second and third columns are the reprojection measures when 3D models are constructed using images segmented with the entropy and Otsu’s methods, respectively. Table 2 shows the result for the light bulb experiment; from the value we can see that the entropy method provides better reprojection than Otsu’s method, implying that the entropy segmentation leads to a more accurate 3D model.
Table 3 shows the result for the computer case experiment. As can be seen from the values, the average reprojection measures for this experiment are lower than the values in Table 3. This phenomenon has also been observed previously with respect to Figure 6, where it was mentioned that the resultant models are not as successful. As a result, the reconstructed 3D model is not as accurate, leading to lower reprojection measures. Nevertheless, in this experiment, entropy also provides better reprojection than Otsu’s method. Table 4 shows the reprojections measures for the metal container. This experiment has been quite successful and the reprojection measures are better than in the previous two experiments. It can also be observed that, like in the previous two experiments, the model constructed from entropy segmentation provides better reprojection than Otsu’s segmentation.
To determine the accuracy of the reconstructed temperature distribution, we measured the temperatures at different locations using thermometers and compare the measured temperatures with the reconstructed temperatures. Figure 8 shows the results of the comparison, where the red and blue dots, respectively, represent the reconstructed and measured temperatures measured at different points on the object. From the figure, it can be seen that there are some locations with larger differences. The reason is that the thermometers are placed in direct contact with the object, where the surface temperatures are often higher than detected by the thermal imager. As a result, the values measured by the thermometer are often higher than the reconstructed values at corresponding locations. Nevertheless, despite the temperature differences, the reconstructed 3D thermal model is still capable of showing locations with higher temperatures within a closed system and provides a general idea of the temperature distribution from external sensors alone.
From the experiments results, we can see that the proposed system is able to reconstruct reasonable 3D models from the images acquired by the thermal imager. Moreover, we can integrate thermal images with the 3D model to estimate the location of the heat source even if it is not visible from the outside of the container. From the reprojection measures, it can also be seen that the entropy approach provides better segmentation results, leading to more accurate 3D models. Therefore, in future experiment we may consider the usage of the entropy approach for segmentation.
5. Conclusions
In this paper a 3D infrared imaging system based on silhouette volume intersection has been proposed to reconstruct the volumetric temperature data. The system uses silhouette volume intersection to reconstruct 3D model of the objects in the thermal images. We compared the entropy and Otsu’s methods for automatic segmentation and use reprojection scoring to determine the accuracy of the reconstructed 3D models. For evaluation, reprojection scoring is used to systematically determine the effectiveness and accuracy of the different approaches. In all three experiments, segmentations using entropy have provided better results than Otsu’s methods.
From the experiment result, we have shown that the system is able to integrate 3D and thermal data to produce a 3D model which has thermal data overlaid upon it, thus providing approximations to the locations of the heat sources. This kind of model is useful when one needs to determine the location of the heat source, which may not be visible, from outside the container; such as in various power related systems, to improve the efficiency in failure detection and the safety of the overall system.
Currently, the proposed method assumes uniform heat distribution between the heat source and the external casing, which may also account for the differences in the measured and reconstructed temperatures. In the future, the method can be modified to cater for cases with different media and include multiple heat transfer functions, such that more accurate reconstructed results can be obtained.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The authors would like to thank the Ministry of Science and Technology of Taiwan for sponsoring this work under Grants nos. MOST103-2221-E-390-021, MOST103-2218-E-110-006, and MOST103-3113-E-110-102. The authors would also like to thank Professor Jen-Hao Teng of National Sun Yat-Sen University, for the use of the thermal equipment.