Abstract
In order to realize the autofocusing in aerial camera, an autofocusing system is established and its characteristics such as working principle and optical-mechanical structure and focus evaluation function are investigated. The reason for defocusing in aviation camera is analyzed and several autofocusing methods along with appropriate focus evaluation functions are introduced based on the image processing techniques. The proposed autofocusing system is designed and implemented using two CMOS detectors. The experiment results showed that the proposed method met the aviation camera focusing accuracy requirement, and a maximum focusing error of less than half of the focus depth is achieved. The system designed in this paper can find the optical imaging focal plane in real-time; as such, this novel design has great potential in practical engineering, especially aerospace applications.
1. Introduction
The image quality is directly related to the optical measurement and reconnaissance equipment measurement result. However, the performance of this equipment can be largely impacted by surrounding environment [1]. For example, because of the change of local temperature, atmospheric pressure, and focal distance from the target, the aerial camera would easily become out of focus [1]; therefore, a robust system which can address this issue is crucial in aerial camera applications. Autofocusing is a key for eliminating the defocusing effect of an aerial camera to ensure high quality of the image. The precision of autofocusing has a direct effect on the imaging quality of the aerial camera. With the improvement of aerial camera’s resolution, autofocusing technique has been greatly desired to improve the overall image quality of aerial imaging [2].
Autofocusing of aerial camera is realized by the following three methods [3–6]. Ground calibration method [7], which is also called program control method: according to the actual measured disturbances, such as temperature, atmosphere pressure, and distance to the ground, which would change the location of the focal plane, this method uses a preinstalled program which is calibrated in advance to adjust the actual position of the focal plane. The advantage of the ground calibration method is that it is simple and is ready to be implemented. However, the practical situation has a certain deviation from the ideal situation, and the ground calibration method only takes into account a limited number of possible disturbances, which could bring in a large error. Optoelectronic autocollimation method [7]: for long focal length aviation cameras with photographic film as a recording medium, optoelectronic autocollimation method is preferred. Optoelectronic autocollimation method can satisfy the requirement of high-precision focusing of aerial imaging. This method can accurately reflect the actual optical system imaging position of infinity target. But the method requires the use of a large, high-quality plane reflector, whose diameter is usually larger than the clear aperture of the camera. Even though optoelectronic autocollimation method is an established technology, this application is too complex to be realized. Furthermore, it can only calibrate the position of infinity target, and it cannot calibrate the position of actual target, which further limits the application of such technique [8–10]. Image processing methods [6], in which the position of the focal plane will be estimated from the actual images: the focusing is an “intelligent” method, which is to say the focusing criterion is flexible, and different focus evaluation function can be automatically chosen according to the requirement. Furthermore, it employs the actual images as the evaluation subjects, so the image processing method is more practical and reliable. In this paper, autofocusing using image processing method is investigated further.
When the aerial camera focuses accurately, the edges and details of the aerial image should be well defined in the spatial domain, and the high-frequency component of the aerial image should be adequate in the frequency domain [11]. To achieve this goal, the differential of image is needed to obtain image edges and detailed information, while for the latter image spectrum information can be obtained by the Fast Fourier Transform (FFT) to convert the image into frequency domain signals. Image differential algorithm can accurately obtain image edges and necessary information, but it cannot filter out the noise of the image. Through image FFT, high-frequency information of the image can be preserved while simultaneously high-frequency noise is cancelled out. However, the FFT algorithm is complicated and time consuming. Aerial systems have to make real-time observations of the targets and, as such, this computationally expensive method is not desirable. The fundamental principle of the automatic focusing system based on image processing method is to count corresponding image focus evaluation function (FEF) value to each focal plane position, and the corresponding position of the maximum FEF value is the best imaging focal plane position [4].
An autofocusing method based on image processing is applicable for processing an image sequence, such as taking images of the target with multiple focal plane positions. According to the focus evaluation function value of image sequence data, the optimal cost function value is found, and the corresponding focal plane position is the best imaging focal plane position. However, due to rapid and continuous movement of aircraft, it is difficult for an aerial camera to take multiple images of the same target. So it is necessary to design an autofocusing method which can obtain the image sequence data to the same ground target with a different focal plane position in real-time. In this paper, an autofocusing method with two CMOS detectors that are located on both sides of the symmetry plane of the imaging CCD detector on the optical path is proposed. The method is applicable to aerial camera with a line scan CCD detector or an area scan CCD detector.
The rest of the paper is organized as follows. The reason of aviation camera defocusing and the theory of autofocusing are analyzed in Section 2. Sections 3–7 introduce the method of autofocusing, including working principle of autofocusing system, soft flow diagram, optical-mechanical structure, the choice of focus evaluation function, and image preprocess. Section 9 gives the experiment result and analysis. Conclusions are given in Section 10.
2. Analysis of Aviation Camera Defocusing
The disturbances that affect the focus of aerial camera such as temperature, atmosphere pressure, and distance to the ground are studied in this section [12].
(a) Defocus Caused by Temperature Change. For single lens, for example, defocus caused by temperature change can be approximately calculated as follows:where is defocus of the focal length caused by temperature change; is focal length of the lens; is temperature variation; is refractive index of glass; is variation under each Celsius; is refractive index of air variation under each Celsius.
(b) Defocus Caused by Atmosphere Pressure Change. For single lens, if the temperature change with height at the same time is ignored, the defocus caused by atmospheric pressure change is given by the following equation:where is defocus of focal length caused by atmospheric pressure change; is atmospheric pressure at certain altitude; is atmospheric pressure when camera is assembled on the ground; is refractive index of air; is refractive index of glass.
(c) Defocus Caused by the Change in Photographic Distance. Defocus caused by the photographic distance is given by the following equation, as shown in Figure 1:where is defocus of focal length caused by photographic distance change; is focal length of lens; is relative height between aerial camera and ground scene; is the angle between camera optical axis and horizontal plane:
As shown in (4), total defocus is the sum of the defocus caused due to individual component. The defocus due to three components can be obtained according to the camera’s optical analysis and simulation results, but the aerial camera in actual use is quite complex. For example, the actual optical system of aerial camera is not a single lens but multiple lens or mirrors, and the lens’ refractive index, curvature radius, thickness, and the gap between the lenses, stress in lens material, and the expansion of metal frame will change due to the change of environment. So it is difficult to understand the focus of aerial camera based on the result from the optical analysis.
The results from the optical analysis are applied to find the focus of lens approximately, which can help in narrowing the scope of focusing and save expired time.
3. Analysis of Aviation Camera Autofocusing
According to optical principle [13], to see a clear image of any object through a thin lens, the Gaussian optics formula must be satisfied:where is the object distance from the lens, is image distance from the lens, and is the focal length of the lens.
The focus adjustment mechanism is to adjust the object distance and focal length such that the Gaussian optics formula is satisfied.
The thin lens model of optical imaging system is shown in Figure 2, where is a dot on the subject and is the image of . When focusing is accurate, and will satisfy (5). When content focuses accurately, the image of light spot size is minimal.
If the camera is not focused accurately, in other words, if the reaction surface of the detector is not at , the image of is a light spot at , it is a blurred image, and its shape is similar to the camera lens aperture. Since the aperture of the camera (especially aviation camera) is circular, the object cannot be focused accurately and the image will form as a fuzzy circle with a radius of on the sensor.
According to the property of similar triangles, the scaling relation between the fuzzy image point and the subject is as follows:
So the radius of fuzzy circle is given by the following expression:
The sensor is located in the clear image plane “,” that is, distance “” from the former position, as shown in Figure 3. According to the properties of similar triangle, the relation between the fuzzy image point and the image point is as follows:
Hence, the fuzzy circle radius is given as follows:
According to the above analysis, when the image detector in the clear image is at the same distance “” before and after , the corresponding blur circle radius will be the same. This shows that the fuzzy degree of two images is the same, and hence, the corresponding focus evaluation function value remains unchanged.
From the result, if one can find two images of different locations and their focus evaluation function value is the same, then the position of the clear image will be in between two positions.
Therefore, this paper puts forward a method of autofocusing with two detectors. Two identical CMOS detectors are used to detect the image on the ground scene for autofocusing, and an imaging detector is located in the middle of the two CMOS detectors.
When the focus evaluation function values of two CMOS detectors on the same ground scenery are the same, that is, the difference between the two values is less than a certain threshold, the imaging detector will capture a clear image as it is located in the clear image position. Thus, the autofocusing will be achieved.
4. Working Principle
Autofocusing system consists of atmosphere pressure sensors, temperature sensors, power amplifier, stepper motor, position encoder, focusing mirror, prefocal CMOS detector, postfocal CMOS detector, and so forth, as shown in Figure 4. Atmosphere pressure sensors and temperature sensors are located on the optical lens of aerial camera and these sensors return the atmosphere pressure and temperature of the aerial camera. According to the relationship between temperature data and focal plane position provided by optical analysis (1), an estimated position of the focal plane is obtained. Atmosphere pressure has a similar impact on the lens and can be analyzed in the same approach. Therefore, the autofocusing controller provides a comprehensive estimated position of the focal plane with atmosphere pressure and temperature inputs. The autofocus controller drives a stepper motor by the power amplifier to move the focusing mirror to an appropriate position.
As forward motion compensation mirror works, the camera controller sends the focus checkup instruction to the autofocus controller. The autofocus controller starts to control and receive the resulting images of the prefocal CMOS detector and postfocal CMOS detector. The autofocus controller captures and averages the gray values of the images from the same detector. After preprocessing, autofocus controller computes the focus evaluation function (FEF) value of these images and drives the stepper motor to move the focusing mirror to the desired position according to the autofocusing algorithm that is shown in Figure 5, until the FEF value of images exported by two CMOS detectors is the same; that is, the difference between two values is less than the threshold value.
As is shown in Figure 6, because the imaging CCD detector is located in the middle of the prefocal CMOS detector and postfocal CMOS detector, according to the characteristic of FEF, the position of the imaging CCD detector is the position where the values of FEF are maximum. Therefore, the imaging CCD detector is located in the best position of the focal plane.
Thus, through the acquisition of the focusing images and control of focusing mirror, the real-time automatic focusing process is realized. The flowchart diagram of the autofocusing system is shown in Figure 5.
This autofocus method combines Phase-Difference AF and Contrast AF. Compared with typical Canon Phase-Difference AF method, this AF method possesses high AF precision and relatively slow AF speed, and it is suitable for low light situations.
5. Autofocusing Algorithm
As shown in Figure 5, autofocus algorithm carries the preliminary focusing according to the surrounding environment, to narrow the autofocusing range.
Firstly, the camera collects the actual temperature, atmospheric pressure, and imaging distance between the aerial camera and target and computes the defocus quantity of the focal plane according to the focal plane position under different condition of temperature and pressure provided by optical analysis. Then the defocus quantity is added to the focal plane position which calibrated in the lab. Finally, a rough focal plane position is obtained. The camera drives the focusing mirror to the approximate focal plane position [10, 14].
General aviation camera has 2 mm range for focusing [15]. According to the camera’s optical analysis and simulation results, the possible focus range will be narrowed down to (1/4)th of the original scope. It shortens the time needed for image inspection method and also for the efficient working of the focus evaluation function [15].
Then, when the scanning mirror works, namely, the camera works in the mode of forward direction image motion compensation (FIMC), ground scenery is relatively stationary to the camera. Now autofocusing controller starts focusing based on image, commanding the postfocal CMOS detector and prefocal CMOS detector image on the ground scenery at the same time. Next autofocusing controller preprocesses the images and then calculates the value of the focus evaluation function of the two images, numerical and . Autofocusing controller compares and , if , and , it is illustrated that FEF value of image obtained by prefocal CMOS detector is bigger than the postfocal CMOS detector, and the best focal plane is near the prefocal CMOS detector. So autofocusing controller drives the focusing lens to move a step backward.
Previous steps are repeated and the sizes of and are compared until . The difference between and is less than 1%; that is, and are mainly the same. As described in Analysis of Aviation Camera Autofocusing above, the best focal plane is in the middle of two CMOS detectors.
Because the imaging detector is placed in the middle of the two CMOS detectors on the optical path, the best focal plane of the imaging detector is in the actual position.
The above work is carried out in front of the aerial camera taking pictures. In the process of aviation camera taking pictures, when the camera is in a state of IMC, focusing controller also controls two CMOS detectors and performs the required numerical calculation. Since the photo has been checked for the focal work before, so at this point the defocus quantity of camera is not big. Generally two to five times of the function comparison can give a best approximate of the actual focal plane; this ensures that the imaging detector is located in the best focal plane, achieving the goal of real-time focusing.
Since this method does not depend on the imaging detector, the line array and area array can be used as imaging detectors. Hence, this method can be adopted universally.
6. Optical-Mechanical Structure of Autofocusing System
The optical-mechanical structure of autofocusing system is shown in Figure 6; postfocal CMOS detector is installed in the optical system after the imaging CCD detector. The distance between photosensitive surface of postfocal CMOS detector and imaging CCD detector is . Before the postfocal CMOS detector, a transflective mirror is placed and prefocal CMOS detector is located in the left side of the transflective mirror. The distance between photosensitive surface of prefocal CMOS detector and imaging CCD detector is also in aplanatic focal plane. The difference is that two CMOS detectors locate at both sides of the imaging CCD detector.
Thus, ground scenery image through reflection of forward motion compensation mirror, refraction of optical lens, and reflection of focusing mirror arrives at transflective mirror. The transflective mirror reflects half of the light to the prefocal CMOS detector, and half of the light is transferred to the postfocal CMOS detector. So the same ground scenery finally images on the two CMOS detectors at the same time.
When the AF CMOS detector is off the optical axis of the system, along with the increase of angle of field, the lens curvature of field will reduce the position accuracy of optimal image plane. So it is necessary according to the astigmatic field curve provided by optical design to place the AF CMOS detectors on the angle of field where the field curvature is minimum.
7. Choice of Focus Evaluation Function
In this paper, the focus evaluation function (FEF) refers to the mathematical expression that is used to obtain the information about image resolution [6]. FEF can roughly boil down to gray level gradient function, the frequency domain function, informatics function, statistics function, and so on. The gray level gradient function is more practical for judging the focusing complexity and stability than other categories functions [3].
The following are four commonly used gray gradient functions.
(1) Gradient square sum function is
(2) Laplace operator sum function is
(3) Roberts gradient sum of squares function is
(4) Secondary polynomial function iswhere , are the column and row numbers of the pixel in spatial domain and is the gray value of the pixel.
For autofocusing system, it is very important to select appropriate focus evaluation function. The ideal focus evaluation function should have the following characteristics: it should be unbiased and have maximum value only when object plane and focal plane overlap; it should have a single-peak: that is, focus evaluation function in the whole range has only one extreme value; it should have high signal-to-noise ratio; in certain noise interference, autofocusing system can accurately detect defocused signal; the computation burden should be as minimum as possible; this is a prerequisite for quick automatic focusing; it should have high sensitivity; that is, autofocusing system should distinguish between focus and defocus; this is an important requirement for focusing precisely [4].
Experiments show that a secondary polynomial function with an approximately linear curvature is suitable for the larger focusing range. But due to its low sensitivity, it is suitable for coarse focusing in a large range. Roberts gradient sum of squares function, gradient square sum function, and Laplace operator sum function are suitable for focusing accurately because of their high sensitivity and high stability in a small range [16].
Comparing several small range focusing functions, Roberts gradient sum of squares function has advantages in terms of minimal computational burden and stability. At the same time, before the application of proposed image-focusing algorithm, an approximate focusing is carried out considering the environmental conditions, so that the possible focal range is narrowed down to a quarter of original. Then, the focus evaluation function which has high sensitivity but fits for small focal range can be used. Hence, the Roberts gradient sum of squares function is chosen as the focus evaluation function. The variation trend of various focus evaluation function is presented in Figure 7.
(a) Gradient square sum function
(b) Laplace operator sum function
(c) Roberts gradient sum of squares function
(d) Secondary polynomial function
8. Image Preprocessing
In automatic focusing of the imaging system, along with the constant change of the camera position and the shaking of view axis or fluctuations of the illumination, average brightness of the received image will change. Since most of the imaging systems possess automatic lighting function, even in the same position, the two-frame image’s brightness is probably different. The difference will affect the output of the focus evaluation function because gray level values of the image are computational basis. The focus evaluation functional values of a clear image with low brightness are less than that of an unclear image with higher brightness. In such cases, focus evaluation functional curve can have multiple peaks. Therefore, image must be normalization in the gray scale.
Figure 8(a) shows 20 frames of images of the same view that are continuously collected. Figure 8(b) shows the corresponding focus evaluation function values. From the experiment results, one can observe the fluctuations in the image; meanwhile, the fluctuations will switch the stability of focus evaluation function. So the image preprocessing refers to 10 frames of the same ground scenery and takes the average of the grayscale values of 10 frames point by point, to reduce the interference mentioned above.
(a) Visible image collection
(b) Focus evaluation functional values
9. Experiment Result and Analysis
In order to validate the performance of autofocusing system, few aerial images in the laboratory are considered. The hardware design of autofocusing system and error analysis are discussed in this section [16].
In order to verify the feasibility of this method, a focus experiment is conducted. The test principle is illustrated in Figure 9, and the material object is shown in Figures 10 and 11. Test apparatus includes simulation air images, optical lens, motorized linear translation stage, one imaging CCD detector and two CMOS detectors for focusing, transflective mirror, autofocusing controller, and monitor computer [6, 16].
The imaging CCD detector is an aerial scan CCD detector, Gazelle GZL-CL-41C6M with a resolution of 2048 × 2048, pixel size of 8 μm × 8 μm, and a frame rate of 30 fps. The optical lens is SMC PENTAX 67 with a focal length of 200 mm, relative aperture of 1 : 6, and an operating wavelength of 0.4~0.9 μm. The focusing detector is an area scan CMOS detector, OV9650, with a resolution of 1280 × 1024, pixel size of 3.18 μm × 3.18 μm, and a frame rate of 15 fps. Autofocusing controller used is S3C6410, ARM11 series processor. KSA300-11/12-X is used as a high-precision electronic linear translation stage with a closed-loop resolution 1 μm.
The target image and the lens are fixed on the desktop. Imaging CCD detector, two focusing CMOS detectors, and a transflective mirror are fixed to electronic linear translation stage. The transflective mirror is placed as shown in Figure 11, to ensure that the two CMOS detectors are in front of the focal imaging detectors and focal plane, respectively, after the same distance “” In the experiment, the autofocusing controller manages the electronic linear translation stage and controls the two focusing CMOS detectors. When the electronic linear translation stage moves to a certain position, two focusing CMOS detectors take photos of the test aerial images. The autofocusing controller preprocesses these images and calculates the focus evaluation function value of images. According to the algorithm presented in Figure 5, the control electronic linear translation stage moves to the next position until focusing ends.
(1) The Experiment Verification of Parameter “” At first, autofocusing controller controls the electronic linear translation stage to move to a certain position till the image CCD detector captures an image clearly. The location is marked as .
To verify this parameter “,” near with a precision of ±0.5 mm, the electronic line translation stage is moved to a random location and the imaging detector is intentionally defocus. According to the proposed focusing algorithm, the autofocusing controller readjusts the electronic line translation stage and controls the two focusing CMOS detectors to capture the image until the end. The location is recorded as .
To compute the parameter “” accurately, a different numerical is selected as shown in Table 1. A city is selected as the test aerial image as shown in Figure 12(b). In Table 1, “” refers to the location where the imaging CCD detector captures the image clearly, which is a location found from the algorithm. “Error” which is given by the difference between the absolute values of “” and “” is computed for each case and results are presented in Table 1. It can be seen from the experimental result that when the parameter “” is equal to 0.10 mm, the autofocusing error is minimum.
(a) River and tree
(b) City
(c) Suburb
(2) Autofocusing Experiment. After finding the parameter “,” three aerial images are used as the target images, as is shown in Figure 12; the above-mentioned experiment method is repeated. The obtained experimental results are shown in Table 2.
When the aerial camera is working, the error of defocus allowed is less than a half of focal depth of the optical system.
It is defined as follows: is the error of defocus allowed; is the reciprocal of relative aperture of optical lens; is central wavelength of optical lens.
According to (14), the test equipment’s error of defocus allowed can be calculated. For an value of 6, the center wavelength of optical system is 550 nm, and the obtained error of defocus allowed is 0.0396 mm.
From Table 2, it can be concluded that the proposed method meets the requirements of the aviation camera focusing accuracy; the maximum error of focusing is 0.029 mm, which is less than the error of defocus allowed, that is, 0.0396 mm. The average error of focusing is 0.017 mm, which is less than a half of the error of defocus allowed, 0.0198 mm. Although three example images are different in content and richness, our focusing method can accurately find the best focal plane.
10. Conclusions
In this paper, a new autofocusing approach of aerial camera was proposed. The proposed autofocusing method is based on the actual image, so it is direct and accurate. Because the ground target is imaged on two CMOS detectors simultaneously, the autofocusing method can reduce the aircraft attitude stabilization accuracy requirements. It can be applied to an aerial camera based on line scan TDICCD detector or area scan CCD detector. When the aerial camera captures the images of the ground scenery, the autofocusing system can work simultaneously. So it is a real-time autofocusing method.
To validate the performance of the proposed autofocusing method, few aerial images in laboratory are considered. Results show that the average error of focusing is 0.017 mm, which is less than a half of the error of defocus allowed, that is, 0.0198 mm. So the proposed method can meet the needs of aviation camera; the error of defocus is within the allowed half depth of defocus. In the future, the autofocusing method based on 2 differential CMOS detectors will be investigated for the better and precise aerial imaging.
Competing Interests
The authors declare that they have no competing interests.