Abstract
Synthetic aperture imaging (SAI) technology gets the light field information of the scene through the camera array. With the large virtual aperture, it can effectively acquire the information of the partially occluded object in the scene, and then we can focus on the arbitrary target plane corresponding to the reference perspective through the refocus algorithm. Meanwhile, objects that deviate from the plane will be blurred to varying degrees. However, when the object to be reconstructed in the scene is occluded by the complex foreground, the optical field information of the target cannot be effectively detected due to the limitation of the linear array. In order to deal with these problems, this paper proposes a nonlinear SAI method. This method can obtain the occluded object’s light field information reliably by using the nonlinear array. Experiments are designed for the nonlinear SAI, and refocusing is performed for the occluded objects with different camera arrays, different depths, and different distribution intervals. The results demonstrate that the method proposed in this paper is advanced than the traditional SAI method based on linear array.
1. Introduction
Synthetic aperture imaging (SAI) uses the camera array to capture the light field of the scene and reconstructs the specified object in the scene by the refocusing algorithm. In this process, the object out of the focal plane will be blurred because of the virtual large aperture of the system.
Object occlusion processing is a big challenge for computer vision, such as object recognition and tracking [1] and de-occlusion imaging [2, 3]. When the object is occluded, the traditional single-view imaging cannot effectively obtain the information of the occluded object. As an important branch of light field imaging, the synthetic aperture imaging technology can overcome the limitation of traditional imaging. It can effectively synthesize the multiview images obtained by the camera array into a virtual large-aperture image. When the detected object is occluded, the synthesized virtual large aperture has a very shallow depth of field, which can effectively blur the foreground occlusion and realize the “perspective” detection of the occluded object (see Figure 1).
(a)
(b)
Camera arrays are the main means to acquire multiview images. Currently, the related theories and applications of SAI mainly focus on the linear arrangement mode, and the nonlinear arrangement mode is rarely used in the field of SAI. When the depth difference between the foreground occlusion and the occluded object is small, the linear array is prone to the problem of serious lack of sampling information of the occluded object, which will affect the reconstruction effect of the occluded object. Therefore, a nonlinear array can be designed by changing the arrangement position of the cameras and the direction of the optical axis to capture more information of the occluded object (see Figure 2). However, the current linear SAI reconstruction algorithm cannot directly adapt to the SAI reconstruction of the nonlinear array.
(a)
(b)
In view of the above questions, this paper proposes a novel approach of SAI based on a nonlinear array and reconstructs the image of the occluded objects. The rest of this paper is organized as follows: Section 2 introduces the related works at home and abroad. Section 3 describes the algorithm, which includes details of nonlinear image rearrangement and SAI of rearranged images. Section 4 presents the experimental results and discussion. Section 5 summarizes the research.
2. Related Works
As we all know, if we need to reconstruct a point in the image, one of the prerequisites is that at least one camera should capture the information about the point. Recently, more and more researchers are working on removing the impact of occlusion. Some of them are interested in image optimization; they use various methods: Yang et al. presented a novel depth-free all-in-focus SAI technique based on the light field visibility analysis [4], the experimental results demonstrate the effectiveness and superiority of the approach; Pei et al. proposed a novel method to improve the image quality of synthetic aperture imaging using image matting via energy minimization by estimating the foreground and the background [5]; Yanyangshuo et al. proposed a refocusing imaging algorithm via foreground labelling [6], the algorithm can effectively remove the information of the occlusion in the scene and improve the quality of the refocused image; Lee et al. presented the concept of a practical free-viewpoint television system with purely optical depth estimation [7]. Although these methods can optimize the reconstruction results, if the camera array cannot capture enough information of the occluded object, it cannot reconstruct the object. Therefore, others studied the arrangement of the camera array after achieving the experiment of SAI by using a linear camera array [8, 9]. Many researchers study simulation of a camera array with a moving camera [10–12]. This method is apparently cheaper than the camera array, and the system affords more flexible camera arrangement. However, the position of the camera is still limited to a plane, and the virtual array formed by the system is still equivalent to a plane array.
When the distance between the object and the occlusion in the optical axis direction is small or the arrangement space of the camera array is limited, the linear array performs poorly in the acquisition of the object information. For this problem, this paper carries out research on SAI reconstruction based on a nonlinear array. Nonlinear arrays can detect a wider range of scene information, and the arrangement of cameras is more flexible than linear arrays. Therefore, nonlinear arrays are widely used for 3D reconstruction of objects, free-view videos of scenes [13, 14], panoramic image generation [15, 16], and scene depth estimation, but nonlinear arrays are still rarely used in the field of synthetic aperture imaging.
3. SAI Algorithm Based on Nonlinear Array
3.1. Rearrangement of Nonlinear Images
Because the optical axes of the multiview images captured by the nonlinear array are not parallel to one another, it brings a great inconvenience to SAI. Therefore, the first step is to rearrange the multiview images. According to computer vision, the one-to-one mapping between the two camera images can be described by the homography matrix H. Relying on the transformation relationship between the images, the captured images can be rearranged so that their optical axes are parallel to that of the reference camera (see Figure 3).
In Figure 3, a point M in space and its projection m in the image are connected by a projection matrix . Their relationship is written aswhere K, R, and t indicate the intrinsic matrix, rotation matrix, and translation vector, respectively.
Because the depth of the occluded object is unknown, we first assume a depth plane which will be reconstructed. By adjusting the assumed depth value, we can get the reconstructed image of occluded objects. In Figure 3, taking the reference camera coordinate system as the reference coordinate system, the X, Y, and Z axes are the horizontal axis, vertical axis, and optical axis, respectively. We assume that the point M with a depth of z located on the optical axis of the reference camera is the point on the reconstruction plane. By calibrating, we can get and (n = 1, 2, 3,…, N, where n ≠ r), which are the rotation matrix and translation vector of the original camera relative to the reference camera, N indicates the total number of cameras, n means the nth camera in the camera array, and r means the position of reference camera in the camera array. The translation vector from each camera to M is , which is defined as
After rearrangement, the position of each rearranged camera is different from the reference camera only on the X axis. The distance between each camera and reference camera, , is defined as
Thus, the translation vector of each rearranged camera to M is
The projection matrix of each camera and the rearranged camera is shown as equations (5) and (6):where K indicates the intrinsic matrix of the camera and E indicates the identity matrix. We make H as a homography matrix from each camera to the rearranged camera, which is defined aswhere and indicate a point on the image of each original camera and the rearranged camera. According to equations (1), (5), and (6), we can calculate H as follows:where is the pseudoinverse matrix of .
When these nonlinear images are rearranged, a lot of object information is lost. To solve this problem, we ensure that each image, and the images taken by the reference camera too, contains enough information about the object by translating the principal point of the images (see Figure 4).
(a)
(b)
(c)
According to equation (7), we can get four vertices of a rearranged image. The new principal point of the rearranged image is derived from the following equation:where represents a set of four vertices of a rearranged image, i = 1, 2, 3, 4.
3.2. SAI of Rearranged Images
According to the traditional linear SAI formula, we can get the following equation:where is the rearranged image captured by the camera n focusing on a depth z, s is the coordinate of a point on the rearranged image, is the coordinate of a point on the rearranged image projected to the depth z, B is the length of the baseline between two adjacent cameras, and is the focus coefficient, which is defined by the following equation:where f is the focal length of the camera (see Figure 5).
From equation (12), we can get the synthetic aperture image , which focuses on depth z. However, the intervals between the cameras are unequal and the principal points of the images are translated; the traditional linear SAI algorithm is no longer applicable. Therefore, to realize the nonlinear SAI algorithm for rearranged images, equation (10) is reformulated as the following equation:where Δs is the translation distance of the principal point and ΔB is the distance between the camera n and the reference camera. Equation (13) can be evaluated to get the synthetic aperture image Sz, which focuses on the depth z. Therefore, the synthetic aperture image focusing on the occluded object is obtained by adjusting the depth z.
Through the above analysis, it is easy to get the flowchart of the algorithm proposed in this paper (see Figure 6).
4. Experiment Results and Discussions
This section describes in detail the experiments conducted to verify the performance of the proposed method. In Section 4.1, a linear array and a nonlinear one are compared for their performance. In Section 4.2, the effect of different distances between foreground occlusion and the occluded object on SAI is studied. In Section 4.3, the reconstruction performance between unevenly distributed and uniformly distributed nonlinear arrays is researched. In Section 4.4, the experimental results are qualitatively and quantitatively analyzed.
In the experiment, this paper uses a single camera fixed on the mechanical arm and an electric turntable to simulate the image acquisition process of the nonlinear array. The electric turntable drives the camera to revolve around the scene. A BASLER camera, model acA1600-20gc, with an image resolution of 1624 × 1234 is used. The electric turntable is 42BYG250Bk-B, produced by RED STAR YANG TECHNOLOGY (see Figure 7).
4.1. Experiment 1: The Comparison between Nonlinear Array and Linear Array
In Experiment 1, the synthetic aperture imaging reconstruction results of the linear array and arc array are compared. The scene information is captured by an arc array (nonlinear array) with a central angle of 80 degrees. The arc array is composed of 41 equally spaced viewpoints with a radius of about 500 mm. The middle viewpoint is used as the reference view, and nine viewpoints are extracted from 41 viewpoints to represent their relative position (see Figure 8). Three scenes are selected for the experiments, that is, three different foreground occlusions and the occluded objects, and the results of synthetic aperture imaging under linear and nonlinear arrays are shown in Figure 9. The occluded objects from the first line to the third line are “Centrum”, “Move Free,” and “云南白药气雾剂”.
(a)
(b)
(c)
(d)
At present, research studies on synthetic aperture imaging at home and abroad are based on linear arrays. The method of Vaish V et al. [3] is a typical synthetic aperture imaging method by using a linear array, and the subsequent studies are based on the method to optimize the image. Therefore, in order to study the effect of the camera array arrangement mode on the synthetic aperture imaging, the synthetic aperture imaging results between the nonlinear method in this paper and the linear method in [3] are compared.
In Figures 9(c) and 9(d), the imaging details are enlarged and displayed in the black box. Because the distance between foreground occlusion and the occluded object is small, the data of the occluded object collected by using the linear array is insufficient and the occluded object cannot be reconstructed well. It can be seen from the imaging results that compared with the method in [3], the method proposed in this paper can display more information about the occluded object.
4.2. Experiment 2: The Effect of Different Distances between Foreground Occlusion and the Occluded Object on SAI
In Experiment 2, the effect of different distances between foreground occlusion and the occluded object on SAI is studied. Two sets of scenes are selected for the experiment: one is a card occluded by a greeting card; the other is a box occluded by a bookmark, and two cones of the same size are placed beside the bookmark. As shown in Figure 10, the length of each object in the scene is 57 mm for the card, 123 mm for the greeting card, 73 mm for the box, 75 mm for the bookmarks, and 79 mm for the cones.
(a)
(b)
(c)
(d)
Experiment 2 and Experiment 1 have the same experimental conditions. The performance of linear SAI and nonlinear SAI is compared by changing the distance between foreground occlusion and the occluded object; the distance is set as 60 mm, 80 mm, 100 mm, 120 mm, 140 mm, and 160 mm. The experimental results are shown in Figure 11.
(a)
(b)
(c)
(d)
(e)
(f)
4.3. Experiment 3: Comparison of Reconstruction Performance between Unevenly Distributed and Uniformly Distributed Nonlinear Arrays
In Experiment 3, the performance of a nonlinear array with uneven distribution and even distribution is researched. The experimental scenario is a box labeled “Centrum” occluded by a larger box labeled “抹茶” and some leaves (see Figure 12). Such scene information is captured by an arc array (nonlinear array) with a central angle of 80 degrees, and the radius of the arc array is 500 mm. The arc array consists of 41 equally spaced viewpoints, which are labeled 1, 2, 3,…, 41 from left to right.
(a)
(b)
(c)
(d)
Experiment 3 and Experiment 1 have the same experimental conditions. In the experiment, 11 unevenly distributed views and 11 evenly distributed views are selected from 41 views (see Table 1). The nonlinear SAI results of unevenly distributed views and evenly distributed views are shown in Figure 13.
(a)
(b)
4.4. Analysis of Experimental Results
In this paper, the peak signal-to-noise ratio (PSNR) and mean structural similarity index measure (MSSIM) are used to quantitatively evaluate the experimental results [17]. The higher the values of PSNR and MSSIM are, the better the occluded objects are reconstructed. The evaluation results of Experiment 1 are shown in Table 2.
It can be seen from Table 2 that the evaluation value of the nonlinear array is higher than that of the linear array. This is because the nonlinear array can capture more information about the occluded object, which can show more details in the synthetic aperture imaging results. The evaluation results of Experiment 2 are shown in Tables 3 and 4; Table 3 shows the evaluation of the card reconstruction results, and Table 4 shows the evaluation of the box reconstruction results.
It can be seen from Tables 3 and 4 that the image reconstruction quality improves as the distance between foreground occlusion and the occluded object increases. This is because when the distance increases, the interference of the foreground occlusion to the occluded object decreases and the information obtained by the camera array increases, which improves the image reconstruction quality. In addition, the nonlinear array can get good reconstruction results especially when the distance between the foreground occlusion and the occluded object is small.
It can be seen from Table 5 that although the number of views is the same, the reconstruction results of unevenly distributed views are better than those of evenly distributed views. Because the uneven distribution array can capture pictures according to the occlusion situation in Figure 8, fewer pictures are captured in front of the object with serious occlusion, while more pictures are captured on the left and right sides with less occlusion, so the uneven distribution array can detect more information of the object and has better reconstruction quality.
5. Conclusions
This paper proposes an algorithm to achieve nonlinear SAI. The algorithm reconstructs the occluded object by nonlinear image rearrangement and SAI of rearranged images. Experiment work on linear arrays and nonlinear arrays demonstrates that a nonlinear array produces better SAI results. Then, the effect of different distances between foreground occlusion and the occluded object on SAI is studied. The experimental results show that the reconstruction performance of the nonlinear array is better than that of the linear array in different distances. Finally, this paper discusses the performance of the nonlinear array with uneven distribution and even distribution. The experimental results show that the quality of reconstruction can be improved by increasing the number of views at the sparse part of occlusion. The future work will focus on synthetic aperture imaging for 3D objects using a circular array.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This research was supported by the Key Laboratory of Transient Impact Technology of China under project no. 614260603030817.