Abstract
In the real vision system, lens always inevitably contains nonlinear distortion, which leads to geometric distortion of digital image, so it must be corrected. In this paper, a nonmetric correction algorithm for lens distortion based on entropy measure is proposed. The algorithm uses the imaging characteristics of the space line in the ideal perspective model, and the distortion entropy is defined to measure the degree of lens distortion. For distortion curves with different distribution, the calculation dimension of distortion entropy measure is uniform, which can reduce the influence of curve inhomogeneity. On this basis, the modified distortion entropy measure with normalized weight is put forward to enhance the capability of noise suppression, and the distortion correction performance of the traditional interior point optimization algorithm, basic artificial bee colony (ABC) algorithm, and Gbest-guided artificial bee colony (GABC) algorithm is compared and analyzed. The simulation experiments demonstrate that the correction performance of GABC to optimize the modified distortion entropy measure with normalized weight is best, and it has strong robustness to noise. Finally, the actual image distortion correction examples verify the effectiveness of the proposed algorithm.
1. Introduction
In the computer vision, vision inspection, image processing, and other research fields, ideal pinhole model is often used to represent camera lens. However, the manufacturing or installation factors in the actual production conditions lead to the phenomenon that the captured digital image has geometric distortion; thus the pinhole camera model is far from enough especially for some low-cost or wide-angle lenses [1]. In order to ensure the accuracy of subsequent vision processing and analysis, the actual distortion of camera lens should be described and corrected.
At present, the algorithm for lens distortion correction can be mainly divided into metric method and nonmetric method. The metric methods [2, 3] require the accurate location of feature points in space, and the distortion model parameters are also integrated into the camera model. These methods establish the corresponding relationship between feature points in world coordinate system and the two-dimensional coordinates in CCD image, and the distortion model parameters and camera model parameters are together obtained by nonlinear optimization. Therefore, these methods have more parameters and require large computation and some known coordinates of feature points; thus the correction cost is higher. In addition, the coupling of distortion model parameters and camera model parameters can easily occur in the nonlinear optimization process, which can affect the distortion correction performance.
The nonmetric correction methods [4] make full use of the geometric invariants of ideal pinhole camera model. Generally, these geometric invariants represent the properties of structures and remain unchanged under appropriate transformations; thus these methods have no need to require some known structure reference. Currently, the available features of geometric invariants include straight line [5, 6], cross ratio [7, 8], vanishing points [9], and plane constraints [10]. The straight line is the most widely used geometric feature, and the projection of straight line should be straight under any perspective transformation. Therefore, [11] uses col-linear feature points to measure distortion, and col-linear vector is adopted in [12]. In addition, geometric feature of straight line also contains line slope [13] and three mutually orthogonal sets of parallel lines [14]. The distortion correction algorithm based on cross ratio invariability is proposed by He et al. [8], which only requires image coordinates of four col-linear points and cross ratio. However, it has some limitations, because only one-order distortion model is considered. On the basis of cross ratio invariability, the linear equations are introduced by Ricolfe-Viala and Sánchez-Salmerón [15], which make the correction method more robust. Zhao et al. [16] have analyzed the accuracy of correction method based cross ratio invariability and propose the distortion correction algorithm using col-linear constraint of vanishing points. Wang et al. [17] use the division distortion model with single parameters, and the distorted image points are adopted for circle fitting; thus this method is limited by the circle fitting accuracy. Reference [18] uses multiview corresponding points to realize distortion correction, but mismatching points can lead to poor correction performance. When the distortion measure has been defined using geometric invariants, the distortion parameters can be finally solved by nonlinear optimization algorithm. The traditional nonlinear optimization algorithm has strong dependence on the initial value, and the setting of initial value has a direct impact on the optimization accuracy, such as Levenberg-Marquardt optimization method and Interior Point Method with constrains. As a research issue in the field of artificial intelligence, the swarm intelligence algorithm has become a new method to solve the traditional optimization problems in recent years. It does not need to set initial value of iteration, and the whole intelligence behavior is carried on through the cooperation of simple individuals, such as genetic algorithm, particle swarm optimization, and artificial bee colony algorithm. Zhang et al. [19] and Chen et al. [20] have in respective successfully applied genetic algorithm and particle swarm optimization algorithm in lens distortion correction. As a novel swarm intelligence algorithm, the artificial bee colony (ABC) algorithm has simple structure, less control parameters, and other characteristic, and it also has better performance in numerical optimization and other application fields [21].
In this paper, the concept of entropy is introduced to measure the degree of lens distortion by using the geometric characteristics of straight line in pinhole camera model. The new definition of distortion entropy has uniform calculation dimension, which can reduce the influence of uneven distribution of distortion curve. Besides that, the distortion entropy measure is further improved to enhance the ability of noise suppression in the analysis of curvature characteristic of distortion curve. The algorithm of ABCs is adopted to carry on the final nonlinear optimization, and the correction performance is compared and analyzed among the traditional interior point optimization, basic ABC, and GABC. In the proposed algorithm, the defined distortion entropy measure is simple and easy to implement, and the modified entropy measure with normalized weight has the noise suppression ability to some extent. In addition, the algorithm of GABC not only has no need to set initial value of iteration but also has better optimization accuracy and faster convergence speed. Finally, a large number of simulation and actual experiments verify the correction performance of the proposed algorithm.
2. Lens Distortion Model and Distortion Measure
The actual camera lens generally has different degrees of distortion due to manufacturing, assembly, and other factors, which can lead to the phenomenon that the actual image points are not identical to the ideal image points, and the difference between them is called lens distortion. Usually, the lens distortion is classified into radial distortion and tangential distortion, and the radial distortion is dominant [2].
2.1. Lens Distortion Model
In general, lens distortion can be quantitatively described using a mathematical model, and the lens distortion model also can be considered as mapping function between actual image points and ideal image points , where
After analyzing the actual imaging process, Brown [22] presented the mathematical expression of radial and tangential distortion model, and it has become the traditional lens distortion model; the formulation is expressed as follows:where , (, ) is the distortion center. , are radial distortion parameters, and are tangential distortion parameters.
However, in practical applications, less parameters cannot fully express the complexity of lens distortion, and too many parameters will also cause the instability of the solution [4]. Therefore, two radial distortion parameters and two tangential parameters are most adopted which can satisfy most lenses with different distortion degree. The model is expressed as
If all the distortion parameters are known, lens distortion can be corrected through the above distortion model.
2.2. Distribution Law of Distortion Curve
If camera lens have no distortion, the imaging projection of straight lines in three-dimensional (3D) space should be straight based on the pinhole camera model and perspective projection principle. In fact, the actual projection of straight lines in space will deform and become distortion curves due to the inevitable distortion. In order to analyze the change law of distortion curves with different distribution influenced by lens distortion, some ideal horizontal lines are generated at intermediate intervals in the CCD image, and equal interval sampling is performed on each line. If the distortion parameters are known, the corresponding distortion points can be calculated by using formula (3). Figure 1 shows points on ideal lines and distortion points on corresponding distortion curve, the solid points represent the points on the ideal line, and the cross points represent distortion points. As can be seen from the figure, if the distance from distortion center to the ideal straight line is closer, the bending degree of the corresponding distortion curve is smaller. Figure 2 shows the relationship between mean curvature of distortion curves and the distance from the distortion center to the ideal lines, and it can be concluded that the mean curvature of distortion curve is greater when it is farther away from the distortion center.
2.3. Distortion Measure
Generally, the ideal imaging points are unknown, so it is impossible to compute distortion parameters according to formula (3). Therefore, the distortion measure is defined to directly describe the proximity between the correction result and the ideal line, and the distortion model parameters are solved by using a nonlinear optimization algorithm to minimize the defined distortion measure. When the correction effect is closer to the ideal straight line, the distortion parameters is closer to the true value. These methods do not need to have accurately manufactured structure reference and belong to nonmetric correction method.
At present, the traditional nonmetric correction methods usually define the sum of square for the distance from the corrected points to the corresponding fitting line as distortion measure. This distortion measure not only requires the fitting of multiple lines but also needs the distance calculation from each corrected point to fitting lines; hence, the computation is relatively large. Besides that, the distribution of distortion curve in CCD and the number of sampling points will also influence the sum of distance square, which make the calculation of distortion measure for each distortion curve nonuniform, then the actual correction effect can be influenced.
Entropy is a very active physical concept in recent years and has been widely used in many fields of natural and social sciences. In this paper, the distortion entropy measure is defined to describe the nonuniformity and bending degree of distortion curve. Because the concept of generalized entropy can usually be defined by the ratio between extensive quantity and intensity quantity, then the intensity quantity of the distortion entropy measure is represented by the distance between the first and the last points of distortion curves, and the extensive quantity of the distortion entropy measure is expressed by the sum of distances between all adjacent sampling points on the distortion curve. The specific formula is as follows: where M, N are in respective the number of distortion curves and the number of sampling points on each curve. represents the corrected point on the distortion curve, and are, respectively, the first and the last corrected point on the distortion curve. As can be seen from formula (4), the calculation dimension of distortion entropy for each distortion curve is uniform, and its value is close to 1, when the camera trends to be ideal pinhole model.
If the coordinates of the actual imaging points contain extraction noise, the distortion model (3) will amplify the extraction noise which may influence the coordinates of corrected points , and the amplification is more obvious when the point far away from the distortion center. In order to reduce the influence of extraction noise, the distortion entropy measure (4) is further improved, which is given by where represents the normalized weight for each distortion curve, which is related to the curvature of the distortion curve, and it can be expressed as In the formula, denotes the mean curvature of the distortion curve.
3. Optimization Method
3.1. Algorithm Description
Artificial bee colony (ABC) algorithm is a kind of swarm intelligence algorithm proposed by Karaboga in 2005 [23, 24], and it has received extensive attention and research from many scholars due to its simple structure, less parameters, and easy implementation. Similar to other evolutionary algorithms, ABC algorithm carries out the iteration search based on a random generated swarm population, and the swarm includes three different kinds of bees, they are, respectively, employed bees, onlooker bees, and scout bees; then the information can be exchanged according to their respective division of labor. Assume that the number of food sources is , one of food sources represents a candidate solution, , and is the dimension of the optimization problem. The food source corresponds to the employed bees, and the number of employed bees is equal to that of onlooker bees. Then, the search process of ABC for the best food source can be described as follows:(1)The employed bees search in the neighborhood of current food sources and generate a new food source; then a better food source should be chosen greedily.(2)Food sources are selected by the onlooker bees according to the information provided by the employed bees, and they search for a new food source in the neighborhood; then current food source should be greedily replaced with a better one.(3)If the food source has not been updated continuously for times, the current food source should be abandoned, and the employed bee becomes a scout bee and randomly searches for a new food source.
The above three phases should be repeated until the number of maximum iterations or the value of allowable error are reached.
In the iterative searching process, the onlooker bees use the information provided by the employed bees to select food source by means of roulette wheel, and the selected probability of each food source is calculated according to the following formula: where is the fitness value of the solution, and the fitness value is used to determine the quality of food source. That is to say, the larger the fitness value is, the better the food source is.
In addition, the fitness function is determined corresponding to the optimization problem. If the optimization problem is to find minimum value, the fitness function is the deformation of the objective function , which is usually expressed as follows:On the contrary, the fitness function is the objective function when the optimization problem is to find maximum problem.
The ABC algorithm performs the greedy selection according to following formula:where is the new food source generated in the neighborhood of the current food source.
In the ABC algorithm, the employed bees and the onlooker bees search for new food source according the following formula: where is a uniform random number in []. is randomly chosen indices, and indicates a random selection dimension.
When the fitness value corresponding to the employed bee cannot be improved continuously for times, the employed bee will become the scout bee, then the scout bee produces a new food source according the following expression: In the above expression, and are the upper and lower bounds for dimension , respectively.
In the basic ABC algorithm, the bee swarm depends on the shared information of individuals to search for new food sources, and its randomness is greater, thus it is generally believed that the ABC algorithm has strong global search capability. In order to enhance the local search ability of ABC algorithm, the global best- (gbest-) guided ABC is proposed by Zhu and Kwong (GABC) [25], and a search operator guided by the best solution is introduced to improve the exploitation ability of the algorithm. Thus, the corresponding search equation is expressed as follows: where parameter is uniformly random number distributed in and is nonnegative constant. Generally, it is set to 2. The remaining parameters are the same as those in formula (10).
3.2. Algorithm Framework
After the description of ABC∖GABC algorithm, the framework of the corresponding algorithm is given below. The ABC algorithm uses formula (10) to generate new food source, and GABC uses formula (12), then the rest of algorithm flow is the same. indicates the current evaluation times of fitness function, and is the maximum number of fitness function evaluations. is used to record the number of times of food source which has not been updated.
ABC∖GABC Framework (1) Generate food sources randomly (2); (3)While do (4)% Employed Bee Phase (5)for to do (6)Generate a new candidate solution according to Eq. (10) or Eq. (12); (7)Update candidate solution according to Eq. (9); (8)if , set , else ; end (9); (10)end (11) Onlooker Bee Phase (12)for to do (13)Choose a food source from the current swarm by the roulette wheel selection mechanism; (14)Generate a new candidate solution according to Eq. (10) or Eq. (12); (15)Update candidate solution according to Eq. (9); (16)if, set , else ; end (17); (18)end (19) Scout Bee Phase (20)if , replace with a newly randomly candidate. (21)if , stop and output the best solution achieved so far,otherwise, go to step (3) (22) Output the best solution.
4. Experiments and Results
In this section, the performance of the technique is assessed using both synthetic and real image data. Since the synthetic image provides exact knowledge of line positions, distortion center, and distortion parameters, the precise quantitative evaluation of performance is possible. The performance on real images is shown to demonstrate the practical implementation of the technique.
4.1. Tests on Synthetic Data
Assume that the resolution of the simulation image is set as , the distortion center () is (380, 280) and the distortion model parameters are, respectively, . Ten simulation ideal lines are randomly generated in the image, and the sampling points are evenly removed on each line, then the corresponding distortion points can be computed based on the distortion model (3). Figure 3 shows the randomly generated synthetic data, and the circle represents the distortion center, the solid points are the sampling point on the ideal line, and the cross point is the corresponding distortion point on actual curve. When the distortion measure function has been defined, the work of lens distortion correction can be transformed into a nonlinear optimization problem which regards the distortion measure as the objective function. In order to reduce the unnecessary search range for nonlinear optimization algorithms, the solution space of the distortion model parameters can be limited as follows: and. The above limitation space can still satisfy the requirement of large distortion image. The simulation experiment is realized on the computer with main frequency 3.6 GHz, memory 8 G, Matlab R2014a, and Windows 7 operating system.
Firstly, the correction performance of the traditional nonlinear optimization algorithm and the ABC global optimization algorithm is compared and analyzed. The traditional nonlinear optimization uses the Matlab optimization toolbox function , which selects the Interior Point Method (IPM) to solve the problem. The initial value for the IPM is set as follows: , and the initial value of distortion center is generally set as the geometric center of the image, that is, . However, the ABC algorithm is a nonlinear global optimization algorithm based swarm intelligence, which do not need to set the initial value, and the control parameters of the ABC are set as follows: the size of swarm population is ; the maximum number of function evaluation is 10000, so the maximum of iterations is 200; because the dimension of correction problem is 6, the control parameter is set as . As the distortion model is a highly nonlinear model, for example, coupling exists between distortion center and distortion parameters, different distortion parameters can cause similar correction results, so it is meaningless to compare the distortion parameters directly [4]. Therefore, the correction performance is evaluated using the root-mean-square (RMS) error between the coordinates of ideal points and those of the corrected points. In order to simulate the feature points extraction error, the Gauss white noise with standard deviation σ is added in the feature points, then the standard deviation is increased by 0.25, and 30 times running is carried on independently at each noise intensity. Finally, the average value of RMS can be obtained.
Figure 4 shows the correction result of IPM and ABC algorithm under the distortion measure , it can be seen from that the correction error increases with the increase of noise intensity, and the correction effect of ABC is much better than that of IPM algorithm. The reason is that the IPM algorithm is a local search algorithm which does not have the global search ability, and it depends on the initial value setting, so it is easy to fall into local optimum value. Therefore, its correction effect is worse. However, the ABC algorithm has strong exploration ability through individual cooperation, and it has been proved theoretically that the ABC algorithm has global convergence [26].
In order to compare the correction performance of ABC and GABC under the distortion measures and , 30 times running is carried on independently at each noise intensity, and then the average value of RMS is also computed. The comparative results are shown in Figure 5. The setting of core control parameters for GABC is consistent with ABC. Besides that, the control parameters of GABC are set as 2 according to [25]. From the comparative results, the following two conclusions can be obtained. Conclusion 1: distortion measure has better noise suppression ability than the distortion measure , because the distortion model will amplify the extraction noise of points and the amplification is more obvious when the point is far away from the distortion center. According to the different distribution of distortion curves, the normalized weights for distortion measure are determined to suppress the noise influence. Conclusion 2: the correction result of GABC is better than that of ABC. Although the ABC has strong exploration ability, its local search ability is insufficient, and it is easy to cause premature convergence. However, the current best solution is integrated into the search equation to improve the local search ability while maintaining the global search ability; thus it can be easier to obtain the global optimal solution for GABC. Figure 6 shows the evolutionary curves of mean value of distortion measure for GABC and ABC at 30 times independent running. It can be concluded that the convergence accuracy of GABC is better than that of ABC, and the GABC also has a faster convergence rate.
To sum up the above simulation results, the correction performance using distortion measure and GABC algorithm is much better. Besides that, it has strong suppression effect on noise and is of more robustness.
4.2. Tests on Real Images
According to the above conclusion from the simulation analysis, we use distortion measure and GABC algorithm to correct actual images. Figure 7 shows the captured checkerboard image. The coordinates of corner points can be extracted by OpenCV function, and its accuracy is 0.1 pixels, then the correction algorithm is carried on to solve the parameters of the distortion model. Table 1 is the obtained parameters of the distortion model. The corrected image is shown in Figure 8, and Figure 9 shows the distortion points and corresponding corrected points, the solid points represent the distortion points, and the cross indicates the corrected points. Table 2 is the analysis data of col-linear points, the Max, Min, and Mean, respectively, represent the maximum distance, the minimum distance, and the average distance from the points to the fitting line. From the comparative result of Figures 7–9, the correction effect is obvious. From the analysis data in Table 2, we can see that all of the distance data after correction is much better than before.
Figures 10 and 11 are, respectively, the scene images and corresponding corrected images. It can be seen that the edge of window and the cabinet are well corrected. In summary, the effectiveness of the proposed algorithm is demonstrated.
5. Conclusion
A nonmetric correction algorithm for lens distortion based on entropy measure is proposed in this paper. The distortion entropy measure is defined to describe the nonuniformity and bending degree of distortion curve, which can reduce the influence of curve inhomogeneity. On this basis, the modified distortion entropy measure with normalized weight is put forward to enhance the capability of noise suppression. Then, the distortion correction performance of the traditional interior point optimization algorithm, basic ABC algorithm, and GABC algorithm is compared and analyzed. The simulation experiments demonstrate that the correction performance of GABC to optimize the modified distortion entropy measure with normalized weight is best, and it has strong robustness to noise. Finally, the actual image distortion correction examples verify the effectiveness of the proposed algorithm.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (U1604151 and 61302118), Outstanding Talent Project of Science and Technology Innovation in Henan Province (174200510008), Program for Scientific and Technological Innovation Team in Universities of Henan Province (16IRTSTHN029), Program for Science and Technology Innovation Talents in Universities of Henan Province (17HASTIT022), the Funding Scheme of Young Key Teacher of Henan Province Universities (2016GGJS-087), and the Fundamental Research Funds of Henan University of Technology (2015QNJH13 and 2016XTCX06).