Abstract
There are some problems in the process of camera calibration, such as insufficient accuracy and poor accuracy. Based on the seagull algorithm, the adaptive differential evolution algorithm is combined with the seagull algorithm to optimize the multicamera calibration. The seagull algorithm can achieve good results on multiparameter problems and effectively avoid falling into local optima. In this paper, the adaptive differential search algorithm is adopted to improve the local search ability and optimize the local search and global search ability. According to Zhang Zhengyou's method, the calibrated parameter is obtained, in which the parameter is used as the initial value. Then, taking the minimum mean error as the criterion, the improved seagull algorithm (SOA-SaDE) is used to establish the objective function, and the internal parameters and distortion coefficient of the camera are further solved. Verification experiments showed that the fusion algorithm has less reprojection error and higher calibration accuracy gull algorithm.
1. Introduction
Multiple cameras are widely used in various fields. Multicamera fusion can provide a wider range of real-time scene information. As an important tool of machine vision, improving camera calibration accuracy is the focus of its research. It has more accurate camera calibration and is more conducive to the camera image matching, recognition positioning, and other follow-up operation accuracy.
At present, the field of deep-sea exploration is developing in a deeper and farther direction. Underwater robots are widely used in underwater exploration due to their wide range of activities and strong autonomy. Compared with other underwater detection methods, robot visual observation is at a close range. Object detection has unparalleled advantages. It can not only track and detect underwater targets in real time but also record underwater video materials for researchers to study deep-sea objects after landing. Therefore, improving the accuracy of camera calibration and eliminating image distortion are of great significance to the research in the deep-sea field.
The camera calibration uses the calibration object in the space and obtains the calibration object different space positions through the position change. The mathematical model can be constructed by the relation between the object space coordinate system and the camera image coordinate system, Then, the internal parameters and distortion coefficients of the camera are calculated and solved.
Camera calibration is a multidimensional nonlinear problem. Each calibration image corresponds to different external parameters. Therefore, it is very difficult to find the concrete calculation equation in the process of backprojection to the calibration point. Common optimization algorithms, such as the pseudo-Newton method, cannot be applied to the optimization of camera calibration because they must depend on the specific functional form. Therefore, the researchers focus on using optimization algorithm to optimize the camera calibration results. Huang et al. [1] used the classical particle swarm optimization (PSO) algorithm to optimize the camera calibration results, took the absolute value of average relative error as the objective function, optimized the camera parameters, and improved the calibration accuracy. Qin et al. [2] proposed a full-parameter adaptive mutation PSO. They used the algorithm to optimize the camera intrinsic parameters and improve the adaptive mutation rate of the particles according to the average particle distance of the particle swarm to optimize the camera calibration results. Xu et al. [3] introduced a diffusion mechanism to improve the local optimal solution problem of particle swarm optimization. Xiang et al. [4] proposed a calibration method based on depth learning. It can be calibrated by inputting the coordinates of the original image by improving the approximation ability of the DNN network. It can be used in large areas, multiple camera angles, and other complex environments. However, deep learning network training requires GPU acceleration and requires high computer configuration, and training takes time, so it is difficult to calibrate quickly. Lei et al. [5] combined PSO with simulated annealing (SA) algorithm, obtained the initial parameters of camera calibration by least squares, and optimized the camera parameters by the hybrid algorithm. This method improves the calibration accuracy of the camera. Based on the Levenberg–Marquardt algorithm, Liu Jiachen [6] corrected the reprojection error by using the improved beam adjustment method to reduce the error of 3D reconstruction. However, in this process, the formula is tedious and the computation is complex.
The seagull algorithm is a kind of metaheuristic, which is suitable for solving multiparameter optimization problems. When the traditional seagull algorithm is used, the initial seagull swarm has strong randomness, but the optimization process is inefficient and easy to fall into local optimization. A hybrid optimization algorithm is proposed in [7]. The algorithm is based on chaotic differential evolution and distribution estimation, which can obtain a high-precision solution. A hybrid algorithm of adaptive gravity search and differential evolution (DE) is proposed in [8], which keeps the diversity of the population. The DE is used for local search and plays a big role. Inspired by the successful application of the above hybrid algorithm, seagull algorithm used in this paper, and seagulls algorithm compared to traditional optimization algorithm, the principle is simple and easy to implement, is suitable for multiobjective optimization, and does not coincide with the position of the population in an iterative process, reducing repetitive iterations and improving the effectiveness of iterations. To avoid the local optimization in the calibration calculation, this paper combines the seagull algorithm and the adaptive differential evolution algorithm and improves the seagull algorithm by absorbing the strong local searching ability of ADE, improving the accuracy and stability of camera calibration.
This article focuses on the key issues that need to be solved for camera calibration. Aiming at the problem of low calibration accuracy and obvious reprojection errors, a fusion algorithm (SOA-SaDE) is proposed based on the seagull algorithm and the adaptive differential algorithm. The internal parameters and distortion coefficients calibrated based on the pinhole camera model are optimized by the SOA-SaDE algorithm. Experiments show that the algorithm proposed in this paper effectively reduces the error of camera reprojection; the reprojection error is reduced by 63.03%, which is 16.75% higher than the effect of the seagull algorithm, and provides a feasible method for reducing the camera reprojection error.
2. Basic Principles of the Seagull Algorithm
In 2018, the seagull algorithm (SOA) proposed a new population-based intelligent optimization algorithm, which simulates the migrating and foraging behavior of seagulls to optimize the target [9].
The seagull algorithm is divided into two parts; they are migration and foraging. Migration is the behavior that is the movement of seagulls from their current position to a more livable position. Migration behavior affects the global exploration ability of the seagull algorithm. Foraging is the behavior of seagulls attacking the food in the current sea area during the flight. The foraging behavior affects the ability of the seagull algorithm for local exploitation.
There are three important points to be paid attention to during a Gull’s migration from one place to another: avoiding collisions between individuals, the best orientation of its position, and its proximity to the best position. To avoid the collision with the seagulls, the algorithm uses the additional variable A to adjust the seagulls’ position:where A represents the migration behavior of seagulls in each given search space The size of A is controlled by B:
The final size of A decreases linearly from 2⟶0 according to the number of iterations. After ensuring that individual gulls do not collide with each other, move all gulls closer to the best:where represents the convergence direction of the individual toward the optimal seagull and B is an important parameter for balancing the exploration and development capability of the algorithm. It changes according towhere rand is a random number in the range [0,1].
After calculating the direction of convergence of each gull, each gull began to move toward this position:where is the position of the seagull.
Seagulls can constantly change their angle and speed of attack during the migration, when attacking prey, seagulls will carry out the spiral movement. The position of the seagull in the 3D iswhere k is a random number at . The algorithm controls the spiral radius by and , and they are usually 1. According to the new position of seagull, the updated formula of the whole position of the seagull is as follows:where is the attack position of the seagull.
SOA: the flow of the algorithm is as follows: Step 1: initialization parameter Step 2: calculate the fitness value for each seagull and the objective function value Step 3: calculate according to formulas (1)–(5) Step 4: calculate according to formulas (6)–(10) Step 5: update position information and fitness values for the best seagull, Step 6: if , skip to Step 7, or slip to Step 3 Step 7: output the optimal seagull position and fitness value
3. Adaptive Differential Evolution Algorithm
3.1. Differential Evolution
Differential evolution algorithm (DE) is a kind of evolutionary algorithm (EA), which is a search strategy for solving polynomial fitting problems put forward by R. Storn and K. Price. This algorithm is based on genetic algorithm and other evolutionary ideas; its essence is to optimize the multiobjective and multidimensional space to achieve the overall optimal solution of the goal. The differential evolution algorithm retains the crossover, mutation, and copy operations in the genetic algorithm. It differs from GA in which the variation vector is generated by the parent difference vector, which crosses with the incidental individuals to generate new individuals and then selects among them. Therefore, it has a better iterative approximation effect than GA. The differential algorithm is divided into two stages: population initialization and iteration [10].
3.1.1. Population Initialization
Suppose that stands for evolution algebra. Then, the 1st individual in the population under the current algebra is represented aswhere D is the dimension of the individuals of the population. In population initialization, the initial population is required to cover the entire search space , and the initialization formula is shown in (8):where is a uniformly distributed random number in the interval and and are the lower and upper bounds of the individual optimization variables , j, respectively.
3.1.2. Differential Mutation Operation
In each iteration, three individual vectors are randomly selected from the population , and , and , according to (13), a new individual can be created; this individual is a variation vector:where F is the variation scale factor, which is used to scale the difference vector to control the search step. In general, the variation scale factor F is in the [0,2] interval.
3.1.3. Cross Operation
In the crossover step, the algorithm adopts discrete crossover. The test vector is generated by crossing mutation vector and target vector according to the binomial method. The specific operation is shown inwhere is the crossover probability factor, and the crossover operator can enhance the diversity of the population. The value of is generally in the interval [0,1], which is a random number uniformly distributed in the interval [0,1]. In the j dimension, if the random generating number is less than , the test vector inherits the variation vector and vice versa.
3.1.4. Select Operation
The selection process selects the more adaptable child from each iteration as the next generation by comparing the child with the corresponding parent based on the value of the fitness function; its selection method is shown as
3.2. Adaptive Differential Evolution
According to formulas (13) and (14), F and are two important parameters in DE, and the choice of their values will affect the optimization effect. However, in the DE algorithm, the values are all constant and cannot be well adapted to various problems, especially, for complex high-dimensional problems. Therefore, Janez Brest introduced adaptive control parameters in 2006. The improved algorithm is called the adaptive differential evolution algorithm (SaDE) [11]. The adaptive control parameters F and are represented aswhere is the random number in [0, 1], τ1 and τ2 indicate the probability of conversion, and and are the boundary scaling factor.
4. Camera Internal Parameter Optimization Design and Application Based on the Hybrid Algorithm
4.1. Design of Hybrid Algorithms
This paper proposes a combination of the SOA algorithm and the SaDE algorithm. It aims to improve the search precision, avoid the population falling into the local extremum, and maintain the population diversity in the later iteration. In the minimization problem, if the fitness of the iteration is greater than that of the previous generation, the location region of the iteration is not good, so the randomness of the population should be strengthened to improve the search range. If the fitness intelligence of the iterated individuals is less than that of the optimal individuals of the previous generation, then the region has the potential value, so we should continue searching along the region.
Combining the SOA algorithm with the SaDE algorithm to realize the internal parameter optimization, in each population iteration, the minimum fitness value of the population is calculated as . In the i+1 iteration, when , SOA is used for optimization, when , SaDE is used for optimization.
4.2. Establishment of the Objective Function
The objective function of the camera calibration problem is established as follows:where is the actual pixel coordinates of the j corner, N is the number of corners, P is the calculated pixel coordinates, are the camera’s internal parameter, are the radial distortion coefficient and the tangential distortion coefficient, and R and T are the rotation translation matrix of the image.
4.3. Parameter Initial Value Solution
The camera imaging relationship is
The above formula represents the transformation of point in the world coordinate system to point in the pixel coordinate system in the linear model. is the internal parameter matrix which represents the intrinsic geometry of the camera. The mathematical model iswhere are the focal length of the camera, and are the physical lengths of the pixels, and and are the intersections of the camera’s optical axis and the image plane.
According to the above expression, the camera internal parameters can mainly solve 4 parameters. They are . We can obtain the initial value of by the imaging relation. The initial value is obtained under the ideal condition, but the actual lens has distortion, so it needs to introduce distortion coefficient to correct it.
The mathematical model of radial distortion is
The mathematical model of tangential distortion is
In the above formula,
Contacting formulas (20)–(22), we can obtainwhere is the image coordinate of the ideal camera model and is the real image coordinate of the nonlinear camera model. The initial value under the distortion is obtained by formula (23).
4.4. Hybrid Algorithm Application
First, the initial value of the internal parameter then initializes the seagull population to generate N different seagull individuals. Initializing population position, parameters A, B, and , set the appropriate parameters = 2, , and initialize the current iteration number t = 0. The fitness function is obtained from the objective function and is defined aswhere (x, y) is the actual pixel coordinates obtained by the corner extraction algorithm, (u, ) is the pixel coordinates calculated by camera imaging relations, and m is the total number of corners.
The algorithm flow of camera internal parameter optimization using the hybrid algorithm is shown in Figure 1.
5. Experiment
5.1. Experimental Design
The experiment uses camera RealSense D435i as the hardware platform and uses Python as the software development platform. In this experiment, 16 images were taken, and the camera internal parameters and distortion coefficients were calibrated based on these images. The pictures are shown in Figure 2.
5.1.1. Specific Calibration Steps
Step 1: the camera calibration is realized based on OpenCV-Python Step 2: according to the calibration parameters obtained by the traditional method, the upper and lower interval of the parameters are set, the scope is limited, and the parameters are initialized Step 3: the results of 300, 500, and 1000 iterations in the seagull algorithm are brought in Step 4: the initial parameters are brought into the SOA-SaDE fusion algorithm to calculate the iterative results of 300 times, 500 times, and 1000 times, respectively Step 5: compare the results of the traditional method and the improved algorithm
5.2. Analysis of Experimental Results
Figures 3(a)–3(d) show the results of 500 and 1000 iterations of the general SOA algorithm and the SOA-SaDE algorithm. As can be seen from the graph that the SOA algorithm converges fast, but it is easy to fall into the local optimum. However, the SOA-SaDE algorithm is effective in dealing with the local optimum and can jump out of the local optimum and continue to converge toward the global optimum. When the image curve is flat, it is close to the optimal target value.
(a)
(b)
(c)
(d)
Table 1 is the calibration result of Zhang Dingyou’s method; Tables 2–4 are the optimization results of the SOA algorithm and SOA-SaDE fusion algorithm after 500 and 1000 iterations, respectively. According to the reprojection error after optimization is calculated, the results of Zhang’s method can be well optimized by these two optimization algorithms. Moreover, the SOA-SaDE algorithm proposed in this paper is generally superior to the SOA algorithm. The fusion algorithm proposed in this paper is robust and reusable and can improve the local convergence of the SOA algorithm and get better results, as shown in Figure 4.
6. Conclusion
In order to solve the local convergence problem, SaDE algorithm is not easy to fall into local convergence. When the optimal individual fitness of each iteration is less than the value of the last iteration, local convergence may have occurred. SaDE was used to optimize population parameters and increase population diversity. This paper presents a new optimization method for camera internal parameters. The algorithm is based on seagull algorithm and adaptive parametric differential evolution algorithm. In this paper, the two are integrated into a framework according to certain mechanisms. By comparing the reprojection errors of Zhang Zhengyou’s calibration algorithm, SOA algorithm, and SOA-SADE fusion algorithm, it can be seen that the gull differential evolution algorithm can get smaller errors. The calibration accuracy is improved to a certain extent. The experimental results show that the gull difference algorithm has good accuracy and feasibility for camera internal parameters optimization. The algorithm can be combined with practical engineering cases to solve multidimensional nonlinear optimization problems accurately and effectively. There are many similar bionic algorithms, such as monarch butterfly optimization (MBO), earthworm optimization algorithm (EWA), elephant herding optimization (EHO), moth search (MS) algorithm, slime mould algorithm (SMA), and Harris hawks optimization (HHO). [12, 13] These bionic algorithms have their own unique characteristics and can play a very good role in specific engineering fields. We believe that, in the follow-up research, we can comprehensively consider the advantages and disadvantages of each algorithm, merge different algorithms, learn from each other, apply it to the vision of underwater robots for deep-sea exploration, and provide a solution for improving the accuracy of underwater robot vision detection. The underwater environment is complicated. How to solve the problem of underwater imaging should consider the refraction of light brought by water, and the problem of floating objects in the water affecting image clarity is the next issue we need to consider.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.