Abstract
In order to solve the problem of panoramic stitching of combined building facade images, the illuminance at the junction is reduced, resulting in the problem of vignetting, the author proposes a vignetting correction method based on 3D laser virtual imaging technology for surface fitting of modular buildings. By calculating the corner points, analyzing the correlation between the image coordinate system and the world coordinate system, obtaining the unit coordinate data of the corner points in the world coordinate system, and constructing a combined three-dimensional interactive model of the camera imaging; The ORB feature matching algorithm is used to perform image matching, by decomposing the homography matrix, the projection transformation matrix, rotation matrix and translation matrix of the camera's internal parameters are calculated, and the cylindrical coordinate projection method is used to complete the panoramic stitching of building facade images; Using the image grayscale correction method based on two-dimensional surface fitting, calculate the correlation coefficient of the Hesse matrix, and use the Hesse matrix to iteratively converge, obtain 2D surface parameter values to eliminate image vignetting. Simulation results show: The vignetting correction time of the combined building facade panoramic image using the vignetting correction method of surface fitting, compared with the common vignetting correction method, the vignetting correction time of the combined building facade panoramic image is shorter, and it is stable at about 5 seconds. Conclusion: The proposed method can effectively correct the vignetting of the combined building facade panoramic image, and the gray distribution of the corrected image is uniform, which shortens the image vignetting correction time.
1. Introduction
In the field of computer vision and computer graphics, the reconstruction of 3D models of buildings has always been a research hotspot; this research direction has a wide range of applications in the fields of smart city construction, virtual reality, and digital protection of cultural relics [1]. For modeling methods that require professional modeling knowledge, they cannot meet the modeling needs of ordinary users. Traditional 3D building modeling methods mainly include procedural modeling, point cloud-based building modeling, and image-based building modeling. Compared with the first two methods, the image-based 3D reconstruction technology, model reconstruction by extracting and calculating 3D information from the input image, it effectively avoids the geometric complexity of 3D scenes, but at the same time there are defects and limitations in geometric accuracy and 3D information integrity, this brings many difficulties to the reconstruction of 3D buildings. In recent years, the research on the reconstruction method of building models based on a single image is more meaningful and challenging than the previous two methods [2]. Figure 1 shows the principle of holographic imaging. In building modeling based on a single image, existing methods mainly use information, such as feature points and feature lines provided in the input image, or the symmetry of geometry and repeated structural constraints [3], by solving the scene camera parameters, the calibration is completed or the 3D information is recovered according to the geometric constraints of the building surface.
2. Literature Review
In the modeling process, the shape grammar in architecture is successfully used in the design and analysis of buildings, and the assembly and construction of buildings are realized through specific grammar rules. With the help of shape grammars, Feng et al. proposed a procedural modeling system for modeling architectural surfaces. According to the building appearance layout given by the user [4], using a heuristic search algorithm, several building models are generated semiautomatically that approximate the given exterior layout [5]. Bui et al. proposed a component-based multilayer parameter modeling method [6], using a probabilistic network to describe the high-level parameters of buildings and components as a user interaction interface, and using a rule base to describe the low-level parameters of components, and combined with the construction template to describe the building construction process. Procedural building modeling using shape grammar usually requires users to have specialized knowledge, and it is difficult to effectively extend the grammar rules. In the building modeling based on point cloud or aerial photography data, the building surface is scanned with a 3D laser scanner to obtain the point cloud data of the building objects, and the point cloud data is processed to realize the 3D modeling of the building. Shan et al. proposed an optimization-based tracking algorithm to achieve 3D modeling of buildings [7]. Based on vehicle-mounted LiDAR scan data, Hizam et al. proposed an interactive SmartBox tool [8], which enables users to construct building models directly on the acquired 3D point cloud data of large-scale urban scenes. Combining aerial data with ground-acquired data, Vu et al. proposed an automatic generation of photorealistic 3D city models with complex surface details [9]. The modeling method based on point cloud or aerial data usually acquires a large amount of data, and the cost of effective data processing is high, which limits the practicality of the method.
In order to improve the imaging quality of the panoramic image of the combined building facade, it is necessary to prevent the occurrence of image vignetting. Thus, the author proposes a vignetting correction method based on surface fitting. Firstly, we construct a 3D interactive model of a combined building, associate the cuboid as a geometric constraint, and obtain the 3D data of the building inside the image [10, 11], we analyze the correlation between the relative corner points in the image coordinate system and the world coordinate system, and then we establish a camera imaging model, IAC representation is used to solve the internal parameters of the camera, so that the visual imaging effect of the combined building facade is better, and the correctness of the panoramic image stitching is maintained. Secondly, the panorama image stitching method is adopted, the ORB matching method is used to extract the adjacent image matching points, the homography matrix is calculated according to the matching points, and the wave correction processing is performed on the camera shake to prevent the image from being deformed, and realize the panoramic stitching of the building facade image. Finally, the surface fitting vignetting correction method is used, and the steepest descent method is combined with the two-dimensional surface fitting to obtain the global change trend of the image gray level, and complete the correction of the vignetting phenomenon of panoramic images. The simulation results confirmed that the author’s method has good applicability and can be widely used in real scenarios.
3. Research Methods
3.1. 3D Interactive Modeling of Modular Buildings
In the process of constructing the 3D model of the modular building, building objects to be constructed are generally constructed from regular cuboid parts. Therefore, the author regards the cuboid set primitive as the basic factor for the construction of the combined architectural model. Using the geometric convergence of the cuboid itself, the three-dimensional data of the building target in the input image can be obtained, and the vertices of the original component cuboid target projected in the two-dimensional image can be marked, it is used in the geometric primitives of the three-dimensional geometry contrast relationship [12]. Due to the large target area of the building, projection into a two-dimensional image often produces distortion, so the corners that need to be marked must cover the six faces of the cuboid.
For the marked six corner points and regions, the goal of camera calibration is to match the cuboid part model and the region-free two-dimensional image contour with each other, after calculating the corner points, analyzing the correlation between the image coordinate system and the world coordinate system, the internal and external parameters of the camera are obtained, and a combined three-dimensional interactive model of camera imaging is constructed. First, the camera imaging matrix is described as the following formula [13]:
In the formula,
In the formula, K represents the projection transformation matrix; R and represent the camera rotation matrix and the translation matrix in turn. The orientation and pose of the camera depend on the parameter sizes of these two matrices. Since there is no distortion behavior when the building is photographed, the distortion parameter s is equal to zero, and the world coordinate system is a symmetrical coordinate system, then , which represents the focal length. We set the coordinates of the principal point to ; according to the homogeneous coordinates, it can be known that the camera projection matrix irradiated to each cuboid vertex and the two-dimensional point in the building image are in a mutual contrast relationship, then
We analyze the dual relationship between the geometric convergence of the parallelepiped on the surface of the 3D cuboid part and the camera’s internal parameters, and we calculate it [14]. represents the aspect ratio of the camera, because , so . We use the IAC notation to solve the camera’s internal parameters, which is defined as the following formula:
Taking the world coordinate system in the scene as the center of the bottom surface of the cuboid, the projection of the cuboid component in the image is realized by the orientation of the camera. We denote the corresponding vertex of the cuboid as the following formula:
In the formula, represents the upper left matrix of L, and and represent the Euclidean morphological data of the cuboid. Assuming that is the homogeneous coordinate of the vertices in the cuboid space, after the camera calibration is completed, the unit coordinate data of the corner points in the world coordinate system can be deduced according to the formula (3), thereby constructing a three-dimensional interactive model of a combined building imaged by the camera, and its expression is the following formula:
The constructed 3D model of the building target can make the combined building facade clearer and more intuitive, and further ensure the accuracy of panoramic image stitching.
The process of constructing a 3D model of a building [15] is shown in Figure 2.
Since the building subject in a single building image captured by the camera occupies most of the area in the image [16], the imaging projected on the two-dimensional image often has perspective distortion, in order to resolve the ambiguity, the user needs to mark the 6 corners that can cover the 6 faces of the cuboid. In the camera calibration, set the origin of the world coordinate system as the center of the bottom surface of the cuboid, and the coordinate axes are aligned with the sides of the cuboid, in order to match the user’s marked area with the two-dimensional projected image area of the cuboid component in the world coordinate system, using the camera model based on the geometric constraints of the parallelepiped, the internal and external parameters of the camera and the size parameters and of the cuboid are calculated.
3.2. Panoramic Stitching of Building Facade Images
In order to stitch a group of continuously shot building images into a panoramic image of building facade, the features of the overlapping area images should be matched first. Here, the ORB feature matching method is used, and the key points are analyzed by binary bit strings [17].
The basic principle of ORB descriptor is to use a small number of grayscale comparisons of image block characteristics to achieve accurate feature matching, with feature points as the center, the criterion for the SxS image block p is described as the following equation:
In the formula, is the pixel gray value of the image block p at the position by smoothing. If the binary feature quantities of 方 orientations are arbitrarily selected in the image block p, the analytical formula of the ORB feature function is the following formula:
Considering many factors, such as speed and precision, the value of o is set to 256.
Key points of measurement using the above process: a 256-bit string of binary bits can be obtained, where each bit has a value of 0 or 1. If and are binary bit strings of two key points, the following formulas
The plane homography represents the projection mapping from one plane to another, and the key includes three parts: the projection transformation matrix of the camera’s internal parameters, the rotation matrix R, and the translation matrix t between the planes.
and in turn represent the spatial coordinates of three-dimensional space points in different viewpoint coordinate systems, and represent the relative matching points of the two images in turn, then
We mark the canonical vector of the plane as n, then all the space points are , because of , equations (12)∼(14) can be obtained:
In an environment with an open field of view, when the camera is rotated to capture images, it can be determined that the camera is doing pure rotation activities, and then the camera is calibrated from a set of overlapping images [18]. First, the homography matrix between all the coincident images is calculated, according to the correlation principle of the plane homography matrix, assuming that the camera performs pure rotation activities, the following formula can be obtained:
According to the orthogonality criterion of the rotation matrix R10, equations (16) and (17) are obtained:where h is the height of the image. Since the image rotation matrix derived from the homography transformation between images is a correspondence matrix, the stitched images will be wavy. In order to make the final stitched image vertical, it is necessary to construct an overall rotating coordinate system.
We reflect the three-dimensional space point to the point in the two-dimensional image, because in the process of capturing the image, the camera will produce irregular motion [19], but usually the horizontal edge of the camera is kept parallel to the ground. In order to build an overall rotating coordinate system, we need to set up a matrix , multiply it by to the right, and make sure that the overall y-axis is perpendicular to the x-axis. We combine the convergence conditions of all images into a set and transform it into a least squares problem:
The author uses the cylindrical coordinate projection method to map the image on the cylindrical surface. The point in the cylinder is determined by the angle θ and the height h parameter, and its relative correlation is the following formula:
Because the cylinder is a surface that can be unfolded, the image can be moved and rotated in the cylindrical coordinates to maintain the same shape, so the panoramic stitching of the building facade image is performed, and its expression is the following formula:
Through the above process, the problem of insufficient viewing angle of the equipment can be effectively solved, a wide-field panoramic image can be obtained, and the real scene of the combined building facade can be fully displayed.
3.3. Realization of Vignetting Correction Based on the Surface Fitting
In the actual scene where the camera captures images, some light beams cannot be imaged at the splicing intersection due to the cover of the mirror, resulting in a decrease in the illumination at the junction, this phenomenon is the vignetting phenomenon formed by the splicing of camera mirrors [20, 21]. Since the reflection edge of the mirror and the optical axis are at an angle of 45°, the distance between the reflection point and the receiving screen is constantly changing, therefore, the vignetting distribution in the image plane also changes with the distance between the mirror and the receiving screen and produce corresponding changes. In addition, straight-edge diffraction is also one of the causes of camera mirror vignetting [22].
Traditional vignetting correction methods include look-up table method, progressive scan method, and function approximation method [23, 24]. The table look-up method needs to use the standard image to obtain the corresponding table of the vignetting coefficient in advance, so the same convergence must be met every time you shoot, and it is not suitable for panoramic shooting of combined building facades. The progressive scanning method uses the real image data to achieve line-by-line fusion, fuses the grayscale change of the full image, and then repairs it; if the grayscale of the two adjacent lines of the real image changes a lot, this method will generate linear fringes, which cannot guarantee the reliability of the final vignetting correction result. The function approximation method is to use multiple calibrations, the compensation elements of a single pixel under each illumination are calibrated, and then the compensation elements of each pixel are fused to obtain the vignetting repair analytical formula of the pixel, but the operation process of this method is relatively complicated.
The author proposes an image grayscale correction method based on the two-dimensional surface fitting, which can effectively improve many deficiencies of traditional methods. Specifically, it is described as the following formula:1
In the formula, , and represent the attenuation rates along the x and y axis of the image, respectively, and is the brightness center point of the reference image, which is also the normalized pixel coordinate; c is a constant offset value. As long as the appropriate x and y axis decay rates are obtained, the direction of its brightness change can be clearly defined, and the gray value of the pixel point, within the vignetting threshold, can also be corrected.
The author uses the steepest descent method to calculate the surface parameters, and obtains the relevant coefficients of the Hesse matrix, the Hesse matrix is used to gradually adopt iterative convergence, and finally the relevant parameter values of the surface are obtained. Based on this, a more accurate parameter estimation value is calculated, which meets the needs of vignetting correction of panoramic images in real scenes.
According to the degraded state of the real vignetting image, we use the two-dimensional curved surface of formula (21) to fit the degraded image. Using the same feature of vignetting as the correction basis, the image brightness attenuation will change with the change of the distance from each pixel to the image brightness center, but in actual operation, the known reference image brightness center is not the image center, so when solving the parameters, the formula (21) should be converted into a Column function, which is specifically defined as the following formula:
In the formula, represents the pixel value of the real image.
Because the gradient of the direct let is equal to zero, the solution process of the equations , , , , and will be very difficult, so the steepest descent method is used to solve the minimum value of F, satisfying , then the gradient of the objective function at point x is described as the following formula:
Because the arctangent function contains the row function, that is, the term, and the Taylor series is used for calculation, then the row function is recorded as formula (24). Considering the complexity of the method, only the first three items are selected to represent the row function.
The above is the analysis of the vignetting correction principle of the proposed method, and it will be used in the actual correction process to simulate the vignetting correction, the calculation process is as follows:
First of all, in a theoretical sense, the steepest descent method can converge on random original values, if the original values are not selected properly, the convergence rate will be too slow, because in the real vignetting correction operation, in order to meet the convergence conditions , , therefore, the original value of X can be approximately selected as , both and are located in the center of the image, and satisfy the correlation of equation (25). In this way, the original value can be iterated, and the convergence time is short, and accurate estimation results can be obtained.
Then, we calculate the values corresponding to the following formulas:
If the corresponding value satisfies the convergence condition, then we end the calculation process and finally complete the vignetting correction of the panoramic image, effectively remove the vignetting phenomenon of the image, and further enhance the image quality. The convergence condition is the following formula:
4. Results Analysis
In order to verify the effectiveness of the vignetting correction method of the combined building facade panoramic image proposed by the author, a simulation is carried out. The experimental environment is a Window10 operating system computer, the dual-core Intel second-generation Core processor OPTIPLEX3010 is the main frequency, the system is Win7 flagship 32-bit, and is implemented by MatlabR2014a programming. The author took aerial panoramic images of building facades as the experimental objects, and the test images were taken on the spot to collect image data, the environmental parameters of the camera used in the experiment are Sony A6000, with 24.3 million pixels, a 3-inch display screen, and 920,000 pixels, and a total of 4 photos were taken.
Due to the influence of camera vignetting, the brightness of the middle area of the data collected by the actual camera is obviously different, the image vignetting phenomenon is more serious, and its gray level change is shown in Figure 3.
Using the vignetting correction method based on surface fitting proposed by the author, the original vignetting image is processed, the surface parameters are calculated by the steepest descent method, and the vignetting surface is fitted according to the degradation state of the real vignetting image, and the fitting results as shown in Figure 4.
Using the surface fitting method, the vignetting surface of the combined building facade panoramic image is corrected, and the correction result is shown in Figure 5.
As can be seen from Figure 5, the overall distribution of image vignetting correction is basically uniform, and the vignetting phenomenon disappears.
Using the method proposed by the author, the vignetting phenomenon of the actual combined building facade panoramic image can be effectively removed, and the grayscale distribution of the corrected image is uniform and the effect is ideal. It can be seen that the correction accuracy of the author’s method is high, and the curve trend is relatively stable, which can meet the high standard imaging of actual panoramic images. It is because the author’s method for vignetting correction, considering the accuracy of panoramic image stitching, has good applicability [25].
In order to further verify the effectiveness of the author’s method, the vignetting correction method of the surface fitting and the ordinary vignetting correction method are used, the vignetting correction of the combined building facade panoramic image is carried out, and the correction time of the two methods is compared, the comparison results are shown in Figure 6.
According to Figure 6, it can be seen that the vignetting correction time of the combined building facade panoramic image using the vignetting correction method of the surface fitting is compared with the common vignetting correction method; the vignetting correction time of the combined building facade panoramic image is shorter, and it is stable at about 5 seconds.
5. Conclusion
In order to effectively enhance the imaging quality of panoramic images of combined building facades, the author proposes a vignetting correction method based on the surface fitting, corrects vignetting in panoramic images of combined building facades. By establishing a 3D interactive model, we can intuitively understand the façade form of the combined building and ensure the accuracy of panoramic image stitching. Using the panoramic image stitching method, we improve the problem of the insufficient viewing angle of equipment and obtain a wide-field panoramic image. We use the surface fitting method to solve the image vignetting phenomenon, shorten the image vignetting correction time, and realize high-quality imaging of combined building façade panoramic images. In the follow-up work, various geometric primitives, such as sphere and cone, can be added to realize more complex combined building model reconstruction.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest.
Acknowledgments
This work was supported by the Jilin Jianzhu University 2021 First-Class Undergraduate Course Construction Project “Advanced Drawing Technology (Architecture) Double Innovation Training, No. 2021050.”