Abstract
This paper is focused on the development and the flight performance analysis of an image-processing technique aimed at detecting flying obstacles in airborne panchromatic images. It was developed within the framework of a research project which aims at realizing a prototypical obstacle detection and identification System, characterized by a hierarchical multisensor configuration. This configuration comprises a radar, that is, the main sensor, and four electro-optical cameras. Cameras are used as auxiliary sensors to the radar, in order to increase intruder aircraft position measurement, in terms of accuracy and data rate. The paper thoroughly describes the selection and customization of the developed image-processing techniques in order to guarantee the best results in terms of detection range, missed detection rate, and false-alarm rate. Performance is evaluated on the basis of a large amount of images gathered during flight tests with an intruder aircraft. The improvement in terms of accuracy and data rate, compared with radar-only tracking, is quantitatively demonstrated.
1. Introduction
Following the hmost important guidelines about unmanned aerial systems (UASs) integration into civil airspace, the onboard avionics of these aircraft must include a sense-and-avoid (S&A) system that is capable to replace the human pilot in performing visual collision threat detection and avoidance. In general, the architectures of detect-sense-and-avoid (DS&A) systems, reported in literature [1–5], are comprised of two subsystems, such as the obstacle detection and tracking one and the collision avoidance one. Several solutions have been proposed to provide the “sense” function. They range from standalone electro-optical (EO) sensors to integrated architectures that comprise airborne radars and collaborative systems based on broadcasted information, such as TCAS (traffic collision avoidance system) and ADS-B (automatic-dependent surveillance-broadcast) [6–12].
A multiple sensor configuration allows compensating single sensor shortcomings as documented in detail in [7, 12]. Thus, in the framework of a research project carried out by the Italian Aerospace Research Center (CIRA) and the University of Naples “Federico II,” aimed at demonstrating a fully autonomous collision avoidance system in flight, an integrated radar/EO configuration was selected. Moreover, in order to satisfy required levels of accuracy, data rate, false alarm rate, and missed detection rate, a hierarchical logical architecture has been considered where the radar is the main sensor and the EO system provides an auxiliary role. This architecture was chosen for several reasons, such as the following;(1)Conventional EO sensors perform initial detection at closer ranges than the radar, even if they are equipped with low noise and high resolution detectors. In some cases these ranges can be shorter than the minimum required to perform a safe avoidance maneuver that is of the order of 2.5 NMi for mid air flight. Indeed, their detection performance is strongly influenced by environmental conditions. (2)EO sensors do not output direct range estimates. Therefore range and range rate information are critical to derive, nonetheless they are needed to guess the probability of collision threats. (3)A standalone radar configuration is not sufficient to provide adequate performance in terms of obstacle revisit rate and accuracy in the determination of obstacle azimuth and elevation angles. In particular, this is true for airborne radars with mechanical scan features. Moreover, radars are affected by performance degradation with adverse weather conditions.(4)Standalone radar tracking estimates can be effective in reducing the computation time and the false alarm rate of EO image processing. Windows with adequate spans depending on the measured range and radar angular accuracy can be selected; they are centered on the obstacle position estimated by standalone radar tracking. (5)In general, sensor redundancy permits to increase overall system reliability and robustness in order to attain required levels of performance in terms of false alarms and missed detections.Thus, the radar sensor performs initial detection and tracking and direct range-to-obstacle measurement offering all-time all-weather coverage, while the EO system permits to increase accuracy and data rate.
In literature EO object detection has been carried out on the basis of different techniques, such as optical flow [13, 14], morphological filtering followed by track-before-detect temporal filtering [15], and morphological filtering augmented with a trained classifier [16]. Generally, these systems require heavy computational resources in order to fulfill real-time full image detection of obstacles. Moreover, their output can suffer from a high false alarm rate since background removal processing is less accurate as the image size increases.
This paper is focused on the description of the obstacle detection techniques adopted to process EO images within the above recalled CIRA project. In particular, it describes the image processing algorithm developed for the panchromatic camera and how we have managed some images critical issue in order to have the best algorithm performance.
The paper is organized as follows. After a brief description of the obstacle detection system architecture, different obstacle detection techniques are analyzed from the application point of view, and the developed algorithm is presented. Technique optimization on the basis of the outcome of flight tests is then discussed. Finally, flight performance of the image processing algorithm is summarized, demonstrating that accuracy and reliability of the developed system are compliant with the requirements of the multisensor tracking system.
2. Obstacle Detection System Architecture
The prototypical obstacle detection and identification (ODID) system was installed onboard the CIRA experimental Very Light Aircraft (VLA) named Flight Laboratory for Aeronautical Research (FLARE) in order to evaluate detection performance in flight tests with an intruder aircraft. It is comprised of a pulsed radar, that is, the AI-130 Obstacle Awareness System model produced by former Amphitech, two visible cameras, panchromatic and color, that is, Allied Vision Technologies Marlin model, and two infrared (IR) cameras, that is, FLIR A40V. The sensor unit is placed on the top of the aircraft wing with the radar in central position. The two visible cameras are installed parallel to the aircraft longitudinal axis, therefore, it is possible to get simultaneously color and panchromatic high-resolution images of the same region. The IR cameras are pointed slightly eccentric to get an azimuth field of view (FOV), that is, comparable to the visible cameras. Radar operates with a 35 GHz carrier which is a good compromise to meet the requirements about antenna dimensions, angular accuracy, and sensitivity to rain and fog. Visible cameras have an FOV of approximately 49.8° × 38.9°, and they output images at the maximum resolution of 1280 × 960 pixels. The panchromatic camera is used for obstacle detection and its output is fused with radar estimate in the Kalman filter [7]. Indeed, the color camera aims at obstacle identification which will be implemented in the future. IR cameras have a smaller FOV of 24° × 18° with a maximum resolution of 320 × 240 pixels. They were installed to detect obstacles in dark luminosity conditions, in order to replace panchromatic camera. Figure 1 shows FLARE with the sensors package installed behind the wing.
Two different processing units complete the overall ODID system: the real-time-tracking computer (RTT-CPU) and the image-processing computer (IP-CPU). The first one is based on a deterministic operating system (OS), that is, Microsoft Windows CE v.5.0, and it is directly connected to the radar via an ethernet link, with Transmission Control Protocol/Internet Protocol (TCP/IP) protocol. It executes the real-time-tracking algorithm and it performs data routing to the guidance, navigation, and control (GNC) system by means of the deterministic controller area network (CAN) bus. The second computer is connected to EO sensors via a Firewire link. It is based on a conventional OS, that is, Microsoft Windows Xp Embedded, and it is dedicated to the processing of visible and IR images. The two processing units exchange data by means of an ethernet link through the User Datagram Protocol (UDP). Figure 2 is a schematic representation of the DS&A system hardware architecture.
Intruder detection by the panchromatic camera is provided following a hierarchical process. Intruder range, azimuth, and elevation as estimated by the radar-based tracking algorithm are the input data to the image processing algorithm. It transforms them from the aircraft body reference frame (BRF) to the camera reference frame (CRF), that is, based on camera alignment matrices estimated by a proper procedure [17]. Subsequently, it checks if they can be included into the camera FOV. If the intruder position is within the FOV, the image processing routine is executed. Only a selected search window is processed in the panchromatic image. It is centered on the intruder position detected by radar, with width and height depending on intruder range and on expected radar and attitude measurement errors. In this way, the object detection algorithm can generate an accurate estimate of intruder angular position in the CRF. In case of detection, this estimate is converted back to the BRF and transmitted to the RTT-CPU. Since only a portion of the whole image is analyzed, the object detection algorithm runs very fast thus minimizing the latency in providing measurements to the multisensor tracking filter.
3. Selection of Image-Processing Techniques
The image processing algorithm selection for the panchromatic camera has followed a detailed comparison of the main techniques listed in literature [18–25]. It has aimed at evaluating the best algorithm in terms of computational load, detection range, false alarm and missed detection rates, and adaptability to various background brightness and illumination conditions.
Several image processing techniques have been considered as possible candidates for the real-time object detection, such as clustering, optical flow, binarization, and coupled edge detection and labeling algorithms.
In particular, the first technique groups the objects together, verifying the internal and external connection of each pixel; optical flow segments a moving object from its background assuming that pixels intensity of local time-varying image regions is approximately constant for at least a short time duration; the binarization technique provides a black and white image characterized by the pixels whose intensity is beyond the settled Gaussian limits; the edge detection—labeling process detects the outlines of an object and the boundaries between objects and background in the image, providing in output a black and white image [19, 20].
The first two image-processing techniques have been discarded from possible candidates for our research applications following theoretical considerations. The clustering technique is computationally heavy thus hindering the real-time performance, and the optical flow is not ideal for our system, because it is characterized by a maximum acquisition data rate of 7.5 fps. In fact, this low date rate provides the pixel displacement speed major than 1 pixel per frame, that is, an optical flow technique requirement [14].
The binarization technique has been applied to several image sequences taken during flight tests, in order to evaluate the best threshold for isolating the intruder aircraft from the background noise. resulted as the best choice for our application on the basis of statistics on images acquired during flights. In the above reported criterion, μ is the mean intensity of the considered search window, while σ is its standard deviation. Assuming a Gaussian distribution of pixel intensity in the search window, this threshold corresponds to an estimated probability of 99,7%. As it will be better clarified in the following, the main drawback of the binarization technique has been demonstrated to lie in its unsatisfying performance for nonhomogeneous background.
Finally, the implementation of the edge detection-labeling technique is carried out following a stepwise procedure, such as(i)building of the search window on the basis of the radar estimates, as already explained in the previous section;(ii)search window binarization by means of the Sobel edge detection technique [18, 20, 21]; in order to limit the impact of background noise, a relatively high threshold has been considered in Sobel method, on the basis of the assumption that the intensity gradient generated by the intruder is larger than the other background objects; more details about threshold selection are discussed in the following;(iii)implementation of the labeling technique which connects all detected pixels in the binarized search window (if any) and outputs a limited number of edges;(iv)detection of the intruder aircraft as the largest edge, that is, the edge that is comprised of the greatest number of pixels. For the sake of concreteness, Figures 3 and 4 illustrate the implementation of the adopted method and its output on a flight image taken during a frontal encounter phase.
(a)
(b)
(c)
Finally Figure 5 shows the edge detection–labeling technique applied on a search window which encloses the intruder aircraft at a range of 2400 metres.
(a)
(b)
Two important points for algorithm implementation are relevant to the dimensions of the search window and the choice of Sobel threshold. They are focused in what follows.
3.1. Search Window Dimensions
Search window definition has to take into account several aspects. First of all, window dimensions have to be set considering uncertainty sources such as radar error in NED and residual time/space registration errors. On the other hand, at relatively small range, the search window has to be large enough to enclose the largest possible obstacles. Finally, window dimensions cannot increase too much because of the consequent computational weight of obstacle detection techniques.
Thus, search window dimensions are defined as follows. When the estimated intruder range is larger than 350 m, the window has constant dimensions in pixels, that is, 150 (width) × 100 (height) pixels, corresponding to an angular FOV of about 6° × 4°. Of course, linear dimensions of the search window increase for increasing range and are always large enough to enclose a typical civil aviation aircraft, such as a Boeing 737 or an Airbus A320. In order to set a lower bound for search window dimensions, when the range is smaller than 350 m the window dimensions in pixels are inversely proportional to range thus achieving constant linear dimensions. Indeed, this case is of few interest for sense and avoid applications.
Figure 6 depicts linear dimensions (in m) of the search window as a function of the estimated intruder range.
3.2. Sobel Threshold
As regards the choice of the Sobel threshold, it is not fixed, but it changes as a function of the estimated intruder range. The most effective thresholds able to distinguish the intruder from the background have been calculated by an empirical analysis based on the large amount of images taken during flight tests [26]. It has resulted that the most suitable threshold applicable at the intensity gradient magnitude increases when intruder range reduces and it changes from about 39 to a maximum value of 42. These results are explained in Figure 7 which reports an example of histogram of gradient magnitude in the search window and depicts the thresholds applied on three different intruder range cases (low, medium and long range). Moreover, the reader can observe that the histogram has a rayleigh distribution, being in accordance with Voorhees and Poggio theory [27].
Moreover, evaluating the probability density function (pdf) of our rayleigh curve, we obtain it is directly proportional to intruder range, standing for the increasing of false alarm probability parallel to the range. However, Figure 8 illustrates the pdf variation in function of the intruder range and it demonstrates that the probability of false alarms is very low, less than 10−8, for the entire threshold slot.
4. Flight Results and Algorithm Optimization
In order to compare the two candidate image-processing techniques, they have been applied to the same sequence of images taken during near collision encounters in the performed flight tests. Indeed, we have obtained the results reported in Table 1. They can be summarized as it follows:(i)the binarization technique has good performance in terms of computational load, false alarm rate, and missed detection rate for brightness conditions of homogeneous backgrounds. Indeed, its performance is degraded for inhomogeneous backgrounds and the relevant detection range of VLA does not overcome 1 km;(ii)on the other side, the edge detection-labeling technique reveals good performance for all requirements, both in terms of detection range and reliability, in fact it is able to detect VLA as far as 2400 m.Therefore, the edge detection-labeling overcomes the binarization in all considered aspects, except for the computational load which is almost equal for both. The computational load and real-time performance was evaluated by means of a laboratory system facility developed to test the sensing system performance [28, 29].
After this first performance estimate, the edge detection-labeling algorithm has been refined aiming at drastically reducing false alarm rate in spite of increasing missed detections. Since the camera output is used to improve the accuracy of the tracker that is based on the radar as primary sensor, it is preferable that the IP-CPU outputs to RTT-CPU fewer, but more reliable EO measurements in order to enhance the overall system tracker performance. In fact, the main effect of a missed detection is to reduce EO measurement rate and thus the algorithm capability to filter sensors noise and improve estimation accuracy, especially on angular derivatives. Instead, if a false alarm falls inside the track gate, disassociation occurs and, due to the small EO measurement covariance, it can have dramatic consequences for tracking reliability such as biases generation and then track loss.
In particular, the critical situations for false alarms risks can be classified in two main categories: presence of sun glares in the captured image, and horizon presence in the search window.
For the first case, a “Sun Presence Detector” has been implemented. It evaluates the percentage of saturated pixels of the whole image and it is based on a statistical study applied on real images. From results, it has been always verified that when sun is in front of the camera, more than 15% of the pixels are saturated. This percentage is the reference threshold considered by the detector; if the sun is detected, the edge detection algorithm is not run and the IP-CPU does not send any EO measurements to tracker.
An example of sun detector implementation is shown in Figure 9.
Horizon presence in the search window is handled in different ways depending on intruder range. In particular, after edge detection-labeling, a further filter is implemented in order to compare the largest edge length to the instantaneous intruder range and to evaluate the probability that it is an intruder or background noise. The reference lengths are the wing span of a civil aviation aircraft (B737) as well as the wing span of smaller vehicles such as the TECNAM P92. They are calculated at different ranges, taking into account also the extension atmosphere and height effects [30], which are modelled by means of the following equation: where is the attenuated length in percentage, is the object range from the point of observation, and is the attenuation factor, depending on the daylight visibility and the object altitude .
However, assuming that flight tests are performed in “haze” environment [30], Figure 10 illustrates the effects of variation of the attenuation factor when altitude increases.
Therefore, the edge detection output is declared as intruder if it is enclosed in the range of reference wing spans. Secondly, the algorithm provides a higher-level control during which it checks if the detected edge has other comparable segments, in this case, it does not output any intruder position measurements. An example of horizon presence in the search window and of the resulting edge detection output is reported in Figure 11.
(a)
(b)
(c)
Applying these controls of false alarm risks, the edge detection-labeling method provides the improved performance of Table 2.
It is important to point out that these results are compliant with tracking requirements expressed in terms of false alarms rate, computational time, and detection range.
Moreover, Figures 12 and 13 show that panchromatic camera measurements allow a great improvement in both accuracy and data rate with respect to radar-only tracking, either for azimuth or for elevation angle estimates. In these cases, there are no EO false alarms and every measurement from panchromatic camera falls properly into the track gate and can be fused with radar estimates by RTT-CPU.
As regards the achievable accuracy and measurement rate, because of the residual uncertainties on data synchronization and sensor alignment, average EO detection accuracy in the North-East-Down (NED) reference frame was found to be of the order of 0.5°, with a measurement rate of 5 Hz, that is, limited by the maximum achievable frame rate at full resolution. Though these values do not represent the performance limit of the EO system, they already show a significant improvement when compared to radar performance (1.7° angular accuracy and measurement rate smaller than 1 Hz on average). It is worth noting that real-time hardware-in-the-loop tests demonstrated an accuracy of about 0.1° [28].
As a result of these performance levels, accuracy and reliability of the developed multisensor tracker can be greatly improved [29].
5. Conclusions
This paper focused on the image-processing algorithm performance for a visible camera which is integrated in a hierarchical sensor architecture for unmanned aerial systems sense-and-avoid applications. The system is installed onboard a very light aircraft and it is characterized by a multisensor solution: the radar, the main sensor, and four EO cameras, aiding sensors to radar, two visible and two IR. In particular, the panchromatic camera aims at increasing the tracking accuracy and data rate after the initial detection by the radar. Therefore, it is mandatory that the EO detection is reliable, fast, adaptable at different weather, and illumination conditions (sunny, cloudy, or rainy day) and it has a large detection range. The most reliable image processing technique for the presented application resulted the edge detection-labeling. It has a computational load adequate to real-time systems, it works well on several image backgrounds and it has a detection range of about 2400 meters in clean air conditions. Moreover, the algorithm has been optimized aiming at obtaining fewer false alarms, in order to increase the overall reliability of the integrated system outputs.
The panchromatic algorithm satisfies the requirements and it has been demonstrated that the overall sense-and-avoid tracking system performance increases significantly with respect to the standalone radar tracking system performance. Indeed, radar can work in all-weather all-time conditions and it performs the direct range-to-obstacle measure, while EO system allows a better angular accuracy, about 0.5° with respect to 1.7° of radar measurements, and an increased data rate, up to 5 Hz instead of 1 Hz provided by the radar sensor.
Acknowledgment
The Ph.D. grant of L. Forlenza was funded by CIRA.