Abstract
Long-wave infrared (LWIR) and visible (VIS) cameras can image information at different dimensions, but the way to calibrate these two types of cameras while registering and fusing the acquired images is difficult. We propose a calibration plate and a calibration method for thermal imaging and visible imaging to solve three problems: (1) the inability of the existing calibration plates to address LWIR and VIS cameras simultaneously; (2) severe heat interference in the calibration images of LWIR cameras; (3) difficulty in finding feature points for registration due to the different imaging spectra between thermal imaging and visible imaging. Simulation tests and error analysis show the error of outline central point computation is less than 0.1 pixel. Average errors of Euclidean distances from the margin outline scattered point sets of the closed circle and closed ellipse to the outline central points decrease by 10% and 9.9%, respectively. The Mean Reprojection Error in the calibration of LWIR and VIS cameras are 0.1 and 0.227 pixels, respectively. Through image registration design and fusion experiments, the FMIdct, MS-SSIM, Qabf, SCD, and SSIM of the images fused after distortion correction are all higher than those of the images fused before distortion correction, with the highest increases being 4.6%, 0.3%, 3.1%, 7.2%, and 1.4%. These results prove the effectiveness and feasibility of our method.
1. Introduction
Camera calibration is the first step in the application of cameras in machine vision. This step determines the internal and external parameters of the visual imaging system. The intrinsic and extrinsic parameters of a camera decide the internal geometric shape and optical properties of its sensors, and its position and direction relative to the world coordinate system, respectively. These parameters ensure the image fusion and collaboration of multiple cameras among machinery visual systems. The existing studies on camera calibration are mostly based on visible (VIS) cameras. Zhang proposed a camera calibration method based on a high-precision calibration plate [1]. As the demand for substance detection in a complex environment is rising, long-wave infrared (LWIR) cameras, which can acquire information regardless of the light source and can detect subsurface information, have attracted wide attention. VIS cameras are often calibrated using black-white chessboard calibration plates or solid circle array calibration plates. Chessboard calibration plates are effective only when a camera is sensitive to visible bands. LWIR and VIS cameras have different spectral bands, and thermal imaging does not show the color features of objects, and its contrast is caused by the specific absorption band within the thermal spectrum. As a result, the variation between the feature space and the nonfeature space of a routine calibration plate is invisible to LWIR cameras at a normal temperature [2, 3]. The calibration methods of VIS cameras are inapplicable to LWIR cameras, indicating specific technology is needed to calibrate LWIR cameras.
The existing calibration methods for LWIR cameras are mostly heatable chessboard calibration plates based on the traditional calibration algorithm. For instance, some researchers used calibration plates consisting of floodlight heating circuit plates, printing chessboard calibration plates, emissivity-specific materials, and bulbs, but the differences in color emissivity led to differences in temperature absorption [4–8]. Large differences in thermal radiance and temperature induced by building a chessboard from two different materials will allow infrared cameras to acquire chessboard patterns [9]. Floodlights cannot uniformly lighten and thus cannot heat uniformly calibration plates. Moreover, the very severe heat interference between adjacent checkers leads to a small temperature-changing gradient and low image contrast between adjacent checkers, which will largely reduce the accuracy of corner detection. Yiu-Ming et al. designed a calibration plate in which metal meshes can be heated in front of plastic plates and used heatgun-heated air as the medium to transfer heat to the calibration plate consisting of plastic plates and metal meshes [10]. But the deformation of metal meshes will reduce the accuracy of corner detection. Hilsenstein made a calibration plate by milling chessboards into copper-foil-pressed plates, in which copper foils had different heating characteristics from glass fibers [11]. The direct contact between copper foils and glass fibers still caused severe heat interference between the two materials, which made the boundary between adjacent checkers blur on infrared images, leading to corner detection inaccuracy.
In this study, a new calibration plate design scheme was proposed. This calibration plate mainly consisted of a black heating plate, a circular hollow white calibration plate, a temperature controller, and baffles. Based on this design scheme, a corresponding calibration method feasible for LWIR and VIS cameras simultaneously was put forward.
The remaining parts of this study are as follows: Section 2 presents an introduction to the overall scheme of calibrating LWIR and VIS cameras, Section 3 elaborates on the calibration plate design scheme, Section 4 presents the calibration method, Section 5 demonstrates an experimental validation, and Section 6 concludes.
2. Overall Scheme
The overall scheme of the calibration method for LWIR and VIS cameras is illustrated in Figure 1 with the calibration of an LWIR camera as an example. The steps are: (1)calibration images are acquired by adjusting the pose of the calibration plate or the camera(2)from the calibration images, feature points for camera calibration computation are extracted, which are exactly the central points of circular hollow pores(3)the feature points as extracted are applied to camera calibration, and the intrinsic and extrinsic parameters as well as the distortion coefficient of the camera are calculated, so camera calibration is finished(4)the distortion coefficient is used to correct distortions in the images collected by the camera thereafter
The LWIR and the VIS camera can both be calibrated following these steps.
The images collected by the LWIR and the VIS camera are registered and fused. After the camera poses are fixed, images of the calibration plate are collected, and feature points are extracted. The feature points of thermal imaging and visible images are matched one-to-one for matrix transformation. The subsequent images are registered using this transformation matrix, and the registered images are fused.
3. Calibration Plate Design
The new calibration plate mainly consists of a black heating plate (Figure 2(a)), a temperature controller (Figure 2(b)), a circular hollow white calibration plate () (Figure 2(c)), and a baffle. The heating plate is regulated by the temperature controller and can heat uniformly. The calibration plate is made from a heat-insulating material, and the heat of the heating plate can only pass the hollow parts of circular pores, but not the solid part of the calibration plate. To solve heat interference, a 25 mm gap is left between the calibration plate and the heating plate, and a layer of heat-reflecting materials is smeared at the back of the calibration plate. The structural schematic of the calibration plate is shown in Figure 2. To meet the demands for camera calibration with different parameters, we design hollow heat-insulating baffles in the same size as the calibration plate, and made the calibration plate into the size of (Figure 2(d)). The LWIR camera capture the heat distribution map passing through the circular hollow array, and the VIS camera directly photograph the calibration plate. In this way, the calibration images can be acquired by different spectral cameras.
(a)
(b)
(c)
(d)
4. Calibration Method
The overall flowchart of the calibration method is shown in Figure 3. First, a region of interest (ROI) is chosen to rapidly detect the calibration plate. Then with the outline fitting method based on least squares, the coordinates in the central points of circular pores are calculated and sorted. The cameras are calibrated according to the sorted coordinates.
4.1. Identification of the Calibration Plate
Before camera calibration, the pose of the calibration plate in the cameras-collected calibration images shall be accurately identified. The traditional calibration method shall traverse the whole image to identify the photographed calibration plate, but the irrelevant background in the image will largely reduce the identification effect. For high-resolution images, whole image traversing will largely waste computation resources. Thus, we create an ROI in the image where the calibration plate pattern is located. The subsequent detection and computation are conducted in the ROI. In this way, the patterns of the calibration plate in the image can be precisely identified and located.
4.2. Detection of the Central Point in the Solid Circle
The circular perspective projection under the camera’s view is often imaged as ellipses, owing to lens distortion, perspective transformation, perspective deviation, and photographing view. Among the existing detection algorithms, the Hough circle detection algorithm requires a high quality of shooting circular patterns in the image [12], and the radius range of the circle needs to be known. Moreover, all the points detected by the edges are detected by drawing circles, and the point where all the drawn circles cross the most is defined as the center of the circle. Then a circle is drawn with this point and the input radius. In the Hough circle transformation, all marginal points are involved in the circle drawing to find out the circle center with the largest weight, indicating the amount of calculation is huge. Thus, this algorithm is inapplicable to the images collected by LWIR cameras.
The central point detection algorithm proposed here is elaborated below.
4.2.1. Edge Extraction and Screening
(i)A filtering operation is used to denoise the calibration image(ii)The adaptive Canny edge detection operator is used to detect circular contours and track the region boundaries in the image. The connected objects with too large or too small pixels in the ROI are eliminated
4.2.2. Calculation of the Central Point of the Solid Circle
During central point computation, the coordinates of the extracted outline points are fitted into a circle (ellipse) through the least squares method, and then the central point of the circle is determined. After that, the distance from each point to the central point in the circle (ellipse) above is calculated. With the difference between the longest distance and the shortest distance, a threshold is set. If the difference is smaller than the threshold, the coordinates of the central point of the circle fitted from least squares are used; otherwise, the coordinates of the central point of the ellipse as-fitted are chosen.
The rationale of the least squares method for circle fitting is shown below.
The constraint of the circular curve equation is;
The difference in the squared sum of square radius in the distance from the extracted outline point to the circle center is;
Let the objective function to be fitted be the sum of squares of ;
The minimum value of can be found when the following equation is met.
The coordinates of the circle center can be obtained by solving the parameters in the above equation.
The rationale of the least squares method for ellipse fitting is shown below.
The constraint function of an elliptic curve is;
Similar to the least squares method of fitting circular contours, the extracted contour points are based on the rationale of the least squares method, and the objective function to be fitted is;
The objective function is optimized to find its minimum value.
The coordinates of the ellipse center can be obtained by solving the parameters in the above equation.
4.3. Coordinate Sorting of the Central Points
Since the outline extraction from the images and the central point detection are both conducted line by line, when the calibration patterns in the images are revolved, the order of detected central points will mismatch with the actual row and column coordinates of the central points in the calibration images. For the convenience of solving, the feature points as extracted shall be sorted according to rows and columns.
4.3.1. Building Linear Equation
As shown in Figure 4, when ROI in the calibration plate is chosen, a quadrangle (the frame surrounded by the red lines) is needed, and a linear equation of boundaries in the tips of the ROI was built. If , no straight slope exists. This straight line is perpendicular to the x-axis (Figure 4(a)), and the linear equation is . If , a straight slope exists (Figure 4(b)), and the equation is, with slope and intercept .
(a)
(b)
4.3.2. Sorting
(i)x-direction sorting. The distance from each central point to the x-direction of the left side straight line is computed and sorted from small to large, forming an matrix. Given the distributive characteristics of the origin, it is unnecessary to calculate the distance from any point to the straight line. This distance can be replaced with the x-direction distance, which improves the algorithmic efficiency(ii)-direction sorting. The number of points in each row is sorted by from small to large, and finally, an central point matrix is formed
4.4. Camera Calibration
In a machinery visual system, to determine the relationship between the three-dimensional coordinates of a certain point on the surface of a real-world object, and its two-dimensional coordinates in the camera-acquired image, we shall build a geometric model of camera imaging (the common camera imaging models are mostly pinhole imaging models). The parameters of the geometric model are exactly the camera parameters, and the process of solving these parameters is called camera calibration.
A point in the three-dimensional space of the world coordinate system (real world) can be expressed as , and a two-dimensional pixel in the image coordinate system can be expressed as . The corresponding augmented vectors are: and . As stated above, an ideal camera geometric model can be regarded as a pinhole imaging model. The relationship between the three-dimensional point in the world coordinate system and the two-dimensional point in the corresponding space coordinate system can be expressed as: where is called the intrinsic parameter matrix of a camera; is its optical center, namely the main point, in unit of pixels; is called the extrinsic parameter matrix of the camera ( is a rotating matrix, and is a translation vector [1]). When the x-axis and y-axis are fully perpendicular to each other, the value of the inclination parameter in is 0; and are the proportion factors on the u-axis and v-axis in the coordinate system of the image. The relative space position relationship between the camera and the calibration plate is illustrated in Figure 5.
Camera distortion refers to the drift of lens-to-line projection. Simply, one straight line in the real world cannot be reserved straight after projection by the camera lens to the image. Camera distortions mainly include radial distortion and tangential distortion. Radial distortion, which includes barrel distortion and pincushion distortion, occurs because the rays farther away from the center of the camera lens are more bent than rays closer to the center (Figure 6). Tangential distortion occurs because the lens cannot be assembled in strictly parallel to the camera sensors (Figure 7).
(a) Normal image
(b) Barrel distortion
(c) Pincushion distortion
(a) Ideal situation lens and sensor are parallel
(b) Real situation lens and sensor are not parallel
(c) Ideal image
(d) Tangentially distorted image
The above camera matrix does not consider lens distortion, since an ideal pinhole model does not have a lens. The real camera lens will inevitably cause distortion, which can be corrected as follows [13]: where are the coordinates in the ideal image coordinate system (the stated above); are the coordinates in the real coordinate system of an image; and are the radial and tangential distortion coefficients of the camera, respectively; .
Distortion and intrinsic parameters are inherent characteristics of a camera and can be always used once calibrated. Since a camera is not an ideal pinhole imaging model and owing to computational errors, the results obtained when calibrating using different images will vary. Generally, the calibration results are acceptable if the reprojection error is low.
As for calibration of a binocular camera, according to the above-calculated results, the positional relationship between the coordinate system of an LWIR or VIS camera and the world coordinate system can be expressed as: where and are the nonhomogeneous coordinates of a point in the coordinate system of the LWIR camera and the world coordinate system, respectively; and are the rotation matrix and translation vector from the LWIR camera coordinate system to the world coordinate system, respectively; and are the nonhomogeneous coordinates of the corresponding point in the coordinate system of the VIS camera and the world coordinate system, respectively; and are the rotation matrix and translation vector from the VIS camera coordinate system to the world coordinate system, respectively. Subtraction of the two equations can discard and yield the positional relationship from the LWIR camera to the VIS camera: where and are the rotating matrix and translation vector of this binocular system. Then the relative position from the LWIR camera to the VIS camera can be expressed as:
The relative position of the VIS camera to the LWIR camera can be determined in the same way. On this basis, binocular calibration of the LWIR and the VIS camera can be achieved [14–16].
5. Experimental
5.1. Detection of the Central Point in a Circle (Ellipse)
The circle (ellipse) central point detection algorithm was validated through simulation experiments (Figure 8). For the sets of ideal closed-circle or closed-ellipse marginal outline scattered points, the outlines fitted from this algorithm are consistent with the real outlines, and the estimated central points overlap with the real central points (Figures 8(a1) and 8(b1)). For sets of isolate arc-like circle or ellipse marginal outline points, the outlines fitted from this algorithm are consistent with the real outlines, and the estimated central points overlap with the real central points (Figures 8(a2) and 8(b2)). During camera calibration, when an edge detection operator was used to find the margin outline, the identified margin outline points were not uniformly distributed as in the ideal situation, because margin diffusion occurred. To test the fitting effect under such diffusion, we introduced some random interfering noise to both the circle and ellipse margin outline point sets (Figures 8(a3) and 8(b3)). After the random interfering noise at weight 0.1 was introduced, from the sets of closed-circle and closed-ellipse margin outline scattered points, this algorithm yielded low errors (less than 0.1 pixels) in the estimated central point from the real central point. After the random interfering noise at weight 0.1 was introduced, from the sets of isolate arc-like circle and arc-like ellipse margin outline points, this algorithm yielded low errors (less than 0.1 pixels) in the estimated central point from the real central point.
Fifteen groups of random interfering noises with the weight from 0.01 to 0.15 at the step length of 0.01 were divided. Under different degrees of noise interference, the x-direction errors, y-direction errors, and Euclidean distance errors between the algorithm-estimated central point and the real central point in the circle or ellipse margin outlines were estimated (Figure 9). As the noise interference degree was intensified gradually, the x-direction error, y-direction error, and Euclidean distance error between the algorithm-estimated central point and the real central point all gradually increased but were all less than 0.1 pixels. These results indicate the new algorithm has high antinoise ability and excellent performance. Figure 10 compares the Euclidean distance errors of the outline central point computed with the proposed algorithm and a clustering algorithm applied to the sets of closed-circle and closed-ellipse margin outline scattered points. The Euclidean distance errors of the outline central points calculated by the proposed algorithm are smaller. The average errors in the Euclidean distance in the outline central points in the sets of closed-circle and closed-ellipse margin outline scattered points decreased by 10% and 9.9%, respectively. Thus, the above simulation results suggest this algorithm can accurately fit circular or elliptical margin outlines and estimate the coordinates in the centers of circular hollow pores.
(a)
(b)
(c)
(d)
5.2. Camera Calibration Test
The types and parameters of the LWIR camera and the VIS camera are listed in Table 1.
The LWIR and the VIS camera were used to collect calibration images from the calibration plate. With the LWIR camera, the heating plate was heated to 40°C before image collection. Twenty groups of correction images at different positions were chosen, and the calculated intrinsic and extrinsic parameters, and distortion coefficients were listed in Tables 2–4.
The Mean Reprojection Error is a commonly-used indicator of camera calibration precision. Reprojection error refers to the difference between the projection (the pixel point on the image) and reprojection (the calculated virtual pixel) of a real three-dimensional point on the image plane. The mean reprojection error refers to the average value of reprojection errors. Figure 11 shows the mean reprojection error of each of the 20 calibration images at different positions acquired by the LWIR and the VIS camera. The average reprojection error of this method during correction of the LWIR camera is 0.1 pixel, and the largest reprojection errors of single images are all below 0.12 pixels (Figure 11). The mean reprojection error during calibration of the VIS camera is 0.227 pixels, and the largest reprojection errors of single images are all below 0.26 pixel. From the images collected by the LWIR camera, the infrared images from the calibration plate show the calibration plate was imaged well with low heat interference (Figures 4 and 12), which was because the calibration plate was preset with a gap and a separation structure. Results indicate the proposed method has excellent performance and can meet the requirements of machine vision for the correction of LWIR and VIS cameras.
5.3. Image Registration and Fusion Test
Based on the above experiments, the geometrical imaging parameters of the LWIR and the VIS camera were both computed, and the results of camera calibration were proved accurate. The binocular imaging system involving the LWIR and the VIS camera can provide complementary features among different spectra.
Since infrared images and visible images are different in both imaging spectra and resolutions, it is difficult to efficiently match these images, which inhibits further data comparison and fusion. Algorithms for registration among visible images can find the feature points between floating images and the reference image and thereby transform the space. But infrared images and visible images are imaging under different spectra, it is difficult to find the feature points between the images. For this reason, based on the calibration plate and circle (ellipse) fitting method proposed here, we can use the following method to register the images collected by the LWIR and the VIS camera from the machine vision system.
In the machine vision detection system, the camera position, and the camera-object relative position are usually fixed, so the camera can be fixed and then used to collect images from the calibration plate. Herein, the proposed algorithm was used to fit each central point in the circular hollow array and sort the central points. The sorted points from the infrared images and visible images were used as feature points for image registration, and the one-to-one relationship was established (Figure 12). After that, these feature points were used to calculate the image transformation matrix. As long as the camera positions and target placement positions were unchanged, these feature points can still be used to compute the image transformation matrix, even if the target substance was not the calibration plate.
The infrared images and visible images were registered. Then the infrared and visible fusing algorithm based on deep features and zero-phase component analysis (ZCA) [17] was used to fuse each group of infrared images and visible images that were matched without or with distortion correction. The specific method is as follows: (1)ResNet is used to extract depth features from visible and infrared images(2)Use ZCA to map deep features to sparse domains(3)The L1 norm is used to obtain the initial weight map. The initial weight is mapped to the size of the source image by bicubic interpolation, and the final weight map is obtained by using softmax associated with the initial weight map(4)The weighted average strategy is used to reconstruct the fused image by processing the source image and the final weighted image
To quantitatively compare the image fusing effects before and after distortion correction, we used six image fusing objective evaluation indices. These include disperse cosine mutual information FMIdct [18], multiscale structural similarity index MS-SSIM [19], gradient-based fusing performance Qabf [20], the sum of differential correlations SCD [21], standard deviation SD [22], and structural similarity index SSIM [23]. A larger value indicates a better performance.
Image fusion experiments were conducted using the example of the hot-melt image and the charger image. The LWIR and the VIS camera were used to collect images of hot-melt temperatures, and the contours and background of the box, respectively. The infrared image and the visible image were fused both before and after distortion correction (Figures 13–1, 2). The LWIR and the VIS camera were used to collect images of charging temperatures, and the contours and background of the charger, respectively. The infrared image and the visible image were fused both before and after distortion correction (Figures 13–3, 4). The image fusion objective evaluation indices were listed in Table 5. The mean FMIdct, MS-SSIM, Qabf, SCD, and SSIM of the images fused after distortion correction increased by 4.6%, 0.3%, 3.1%, 7.2%, and 1.4%, respectively, compared with the images fused before distortion correction. The effects of all groups fused after distortion correction was improved in comparison with those fused before distortion correction, except for SD in the first and third groups. The infrared image and visible image fused after distortion correction outperformed the infrared image and visible image fused before distortion correction.
6. Conclusions
A multispectral camera calibration plate for both LWIR and VIS cameras was designed. Based on this correction plate, a camera calibration method was put forward. Specifically, the calibration patterns of infrared images and visible images were detected by selecting ROI. Then the center of circular holes was fitted using the least squares method, and its coordinates were calculated. Even under noise interference, the coordinate error was still less than 0.1 pixel. The average errors of Euclidean distance from the outline central points to the margin outline scattered point sets of the closed circle and closed ellipse decreased by 10% and 9.9%, respectively. An algorithm that sorted the center point coordinates according to the tip of ROI was proposed and used in subsequent camera calibration computation. The coordinates at the center of circular hollow pores were calculated and sorted, and then the intrinsic and extrinsic parameters and distortion coefficients of cameras were computed, which thereby achieved camera calibration. LWIR and VIS camera correction experiments were conducted to validate the calibration plate and the camera calibration method proposed here. The Mean Reprojection Errors in the 20 calibration images collected by the LWIR and the VIS camera were 0.1 and 0.227 pixels, respectively. To further validate the camera calibration results, we designed image registration and fusion experiments based on the LWIR camera system and the VIS camera system. The FMIdct, MS-SSIM, Qabf, SCD, and SSIM of infrared images and visible images fused after distortion correction increased by 4.6%, 0.3%, 3.1%, 7.2%, and 1.4%, respectively, compared with the images fused before distortion correction. Hence, the calibration plate and the camera calibration method proposed here are effective and efficient.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Authors’ Contributions
Sicong Sun and Wei Wei are the co-first authors and they contributed equally to this work. Rougang Zhou is the corresponding author.
Acknowledgments
This work was supported by the Mstar Technologies, Inc.