Abstract
This paper considers real flight application of a monocular image-based aircraft collision decision algorithm developed in a previous paper. First, it briefly summarizes the theory based on the previous paper and extends the results with a detailed evaluation of possible special cases. Second, it introduces the UAVs and flight test scenarios together with the camera system and the steps of image processing used in flight testing. A brief analysis about intruder detectability is also provided referencing a more exhaustive work of the authors. The main contribution is the detailed comparison of the image-based estimated collision parameters to the flight trajectory-based ones together with threshold selection for collision decision. Different threshold combinations are evaluated offline, and finally, real flight decision results based on one of the threshold combinations are also discussed. The paper ends with the setup of future research directions.
1. Introduction
Sense and avoid (S&A) capability is a crucial ability for future unmanned aerial vehicles (UAVs). It is vital to integrate civilian and governmental UAVs into the common airspace according to [1] for example. At the highest level of integration, airborne sense and avoid (ABSAA) systems are required to guarantee airspace safety [2].
In this field, the most critical question is the case of noncooperative S&A for which usually complicated multisensor systems (see [3–6] for example) or radar-based solutions (see [7, 8] for example) are developed. However, in the case of small UAVs, the size, weight, and power consumption of the onboard S&A system should be minimal. Monocular vision-based solutions can be cost and weight effective and therefore especially good for small UAVs [9–16]. These systems measure the position (bearing) and size of the intruder aircraft (A/C) camera image without range and physical intruder size information. This scale ambiguity makes the decision about the possibility of collision (collision means both mid-air collision or near mid-air collision in this article) complicated (see, e.g., [9] where a conservative assumption on intruder’s minimum size is used to make the decision possible but this can lead to false alarms). In another source [13] where the task is to follow a leader aircraft based on a monocular camera, special maneuvers are required to provide range observability. However, [10] points out that the relative distance of the intruder from the own A/C called the closest point of approach (CPA) well characterizes the possibility of collision together with the time to closest point of approach (TTCPA) and there is a chance to estimate these parameters without any physical intruder information. In the literature, there are several works dealing with TTCPA estimation based on optic flow or first-order motion models such as [17–19]. However, it is hard to find references about the estimation of CPA. For example, [9, 20] target to estimate the absolute distance between the A/C the former with the conservative intruder size assumption and the latter by assuming known intruder position and relative velocity. Consequently, previous efforts of the authors [21–24] focused on the estimation of TTCPA and CPA values together with the direction of CPA from solely the monocular image parameters. The only restrictive assumption was that the aircraft follows straight flight paths with constant velocities (in these previous works, TTCPA was sometimes referenced as TTC (time to collision)). Formulae to obtain these values with forward looking and even with oblique cameras (see our system in Figure 10) were developed and tested mainly in software in-the-loop (SIL) simulations. The proposed simple solution is based on a least squares (LS) optimal line fit method which well serves real-time execution needs. Error analysis considering image pixelization errors was done in [21].
Our latest work [24] presents the theory of evaluation of the collision situation in three dimensions (3D) estimating TTCPA, horizontal and vertical CPA, and the direction of the horizontal one () as it is not necessarily perpendicular to the own A/C forward body axis (see [23] and Figure 2). The only information lost is the absolute distance between the two A/C; the relative CPA values are determined instead, but this information is appropriate for the decision about the possibility of collision and for the design of an avoidance maneuver. SIL test results are presented, but in-flight testing of such methods is also very important as underlined by [4, 13] for example.
So, the main goal of the current article is to test the developed method on real flight data. That is why after a summary of the theory published before (to make this article self-contained) and the additional evaluation of special cases, it introduces our real flight test system (A/C and camera system) and applies the derived 3D collision characterization on real flight image data. Knowing the sizes and real flight GPS trajectories of the A/C, the obtained camera-based results (TTCPA, CPA, and ) are compared to the real data. Finally, suggestions are done for decision threshold selection and real flight decision results based on one possible combination of the thresholds are evaluated. The structure of the paper is as follows. Section 2 briefly summarizes the 3D collision characterization from our previous article [24] by possibly partly reproducing the wording used there. Then it includes detailed evaluation of the possible special cases. Section 3 introduces our test UAVs and the test scenarios. Section 4 presents our camera system together with the image processing issues. Section 5 summarizes the results of collision situation evaluation in real flight and presents some results about intruder detectability. Finally, Section 6 concludes the paper.
2. Theory of 3D Collision Possibility Decision
This section includes a summary of the results obtained in [24] estimating first the horizontal closest point between the A/C and the related vertical distance () and second the vertical closest point and the related horizontal distance (). These points are shown in Figure 1 together with the own aircraft body coordinate system. The dashed lines symbolize the aircraft flight paths. Note that the order of body coordinate system axes differs from aerospace convention to better match camera coordinate system conventions of the image processing community.
Considering the interpolation between and , the real 3D minimum CPA can be obtained by applying the normalized equivalent of the formula presented in [20] equation (E4).
The underlying assumption of all of the derivations is that the A/C fly on straight paths with constant velocities. The collision situation is evaluated relative to the body coordinate system of the own A/C. The separate camera coordinate system will be used to characterize camera orientation relative to the body if it is not aligned with the body system (see, e.g., Figure 3).
The horizontal situation in the - plane of the own body coordinate system is shown in Figure 2 denoting also , the horizontal closest point. To evaluate this situation (obtain ), one has to determine , , and . Here, subscript denotes that it is a horizontal parameter, while the second subscript denotes that the parameter is determined by looking for the horizontal minimum distance. is the absolute minimum horizontal distance while is the characteristic horizontal intruder size.
In the horizontal plane, the A/C can be modeled as a disc in the same way as in [9]. Considering that the camera of the S&A system can be oblique relative to the body coordinate system (see our system in Figure 10), the disc projection model of the sense and avoid situation is shown in Figure 3. For a detailed derivation of the disc-based camera projection model for an oblique camera, see [22].
Finally, a system of equations (1) results which includes the modified image parameters and . Where is the horizontal position and is the horizontal size of the intruder image, is the camera focal length, is the camera angle, and and are the view angles of the edges of the intruder image as shown in Figure 3. All are measurable parameters in the image and , are known. The other parameters , , and and the constant intruder relative velocities are unknown (note that is shown in Figure 2 and its estimation will be discussed later).
In the vertical situation, the unknown parameters to obtain decrease to , as the relative angle will be surely 90° up or down. In the vertical plane, the A/C cannot be modeled as a disc since it is rather similar to an ellipse (length almost equal to the horizontal size of the intruder and height equal to the height of the intruder). However, the pinhole camera projection model of the ellipse is very complicated and later interpolation for the 3D minimum point requires normalization with the same size in the horizontal and vertical CPA. Thus, an extension to the same disc size as in the horizontal plane is done considering also the distortion effect dependent on the angles. This is shown in Figure 4. For details, see [24]. Finally, the extended vertical size in the image plane can be denoted as .
This leads to a similar system of equations as in the horizontal case (see (1)) with modified image parameters and .
The first equation is exactly the same as that in the horizontal plane so the mean value of the two equations can be considered. Summarizing the results up to now leads to three image parameter-based equations from which the first two characterize the horizontal situation () while the last characterizes the vertical situation ():
In this system of equations, the unknowns are , , , and . The terms such as the camera angle and focal length , the relative velocities , and the intruder size are all constant. Considering this and , one gets another form of the equations in (4) and (5). Note that is the actual varying absolute time while and are the constant absolute times when the intruder is closest to the own A/C in the horizontal and vertical planes, respectively.
The horizontal minimum and the related can be determined from (4). Fitting LS optimal linear curves to the expressions on the left side with independent variable gives , and this leads to a system of linear equations for and :
This system gives a solution for and considering the crossing of the body-fixed -axis by the intruder as the closest point. However, as Figure 2 shows, this is not necessarily the closest point. Perturbing the camera angle to get (this means virtual rotation of the body system), one can derive the optimal value of which gives real minimum and the related and in the horizontal plane.
Turning back to the vertical situation from (5), that equation characterizes CPA in the vertical plane. In the case where , and can be determined simply from the time average of values. For this case, the A/C flight paths are parallel in the vertical plane. In the other case, and mean that will be the closest vertical point as the trajectories cross each other in the vertical plane. For this case, . Doing a line fit to the data pairs will give and from which can be determined: .
It can be proven that if and , the closest horizontal point will also be the closest 3D point (for details, see [24]). Its coordinates result as follows (where stands for the vertical parameter and for the horizontal parameter):
In the other case, if , one should determine the vertical position of the intruder () in point while the horizontal position is at , , and and also determines the horizontal position of while the vertical position is at and .
In , can be determined from (5) by substituting into obtained from LS optimal line fit. In , and the related (and so ) values should be determined. This can be done considering the system of equations for the averaged image size and horizontal data from (4) and (7) ().
As is known, the system can be reformulated having the known values on the right-hand side: where the unknowns are and . Depending on the values of and , there can be different solutions. For details, see [24].
Summarizing the developments, one has two points, with parameters and with as shown in Figure 1.
The coordinates of points and in the body coordinate system ( in Figure 1) so result as follows:
An interpolated point between and can be represented as follows. Note that the given representation with the parameter makes extrapolation before or after also possible.
Minimizing the 3D distance gives which is exactly the same as the value calculated in [20] (E4) from absolute trajectory parameters, but here, this is an extension of the formula relative to CPA distances. Considering this result, the time of minimum 3D distance and the related relative CPA values in case the A/C trajectories cross the vertical plane can be determined as follows:
Finally, related to the real vertical size ( real vertical image size) of the intruder should be calculated from (calculated from the extended vertical image size) considering the ratio between horizontal and vertical sizes and taking its time average :
The approximation in (15) shows that there can be different distortions of and . A method to correct this is proposed in [24] and so should be calculated based on the corrected values.
2.1. Possible Special Cases
After developing the formulae, they should be examined for special cases to avoid any singularity in the calculations. Sources such as [25, 26] point out that a critical special case in A/C collision avoidance is the case of collision with zero miss distance where the intruder A/C will be steady in the pilot’s view. For this case, it will be also steady in the camera system’s view and so this can be a problematic case in image-based S&A also. Considering an oblique camera setup, three special cases can occur in the horizontal plane:
Case 1. , , and which means a head-on scenario with zero miss distance and an oblique observer camera.
Case 2. , , and which means a crossing scenario with zero miss distance and a forward-looking camera.
Case 3. , , and which means a head-on scenario with zero miss distance and a forward-looking camera.
These cases are examined in the sequel considering also the possible vertical situation and that in all cases because .
2.1.1. Case 1 ()
In this case, the horizontal equations from (3) and (4) degrade to
This shows that the LS optimal line fits can be done flawlessly and so are obtained with and theoretical values. Substituting these into the expression of in (7) gives
This gives and the following form of the equation system to be solved from (7) (considering also and theoretical values):
It is easy to derive the solution as which gives and . Considering from (7), one gets as and .
Considering now the possible vertical situations if , it is either or if the A/C fly in the same horizontal plane . The calculation of both of them is straightforward from the mean value of and will be the same in the further two cases also. The only vertical scenario left is the crossing of the own horizontal plane by the intruder with and . For this case, after determining , should be determined from (10) considering that , . The relation of and gives three possible cases shown in Figure 5.
The vertical crossing can be earlier than the horizontal one (in front of the own A/C) (; no. 1 in Figure 5), and it can be at the same time (; no. 2 in Figure 5) or later (behind the own A/C) (; no. 3 in Figure 5). In the second case, , , and according to [24]. In the other cases, and so the formula from [24] can be used to determine
Substituting the theoretical values of , into , and the latter into (20) equation gives
This gives and can be calculated from this. Finally, which is either or radian giving the expected in-front or behind crossing.
Finally, the calculation of the 3D minimum distance with interpolation should be examined for the special case. The final parameters are , , , or , and ( can be determined from (5) knowing and ). This gives the coordinates of points and as and , , from (11) and (12), respectively. Considering the interpolation, the only singular case is where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing parameter with , , and .
2.1.2. Case 2 ()
In this case, the LS optimal line fits can also be done flawlessly and so are obtained with and theoretical values. Substituting these into the expression of in (7) gives
This is the calculation formula that resulted from a forward-looking camera in [23] so it is natural to get this result. Now, the equation system to be solved from (7) results as (considering also , ) giving again which gives and . Considering from (7), one gets as . This is only a theoretical value because and has no definite direction.
Considering the only special vertical case with and , no. 1 and no. 3 cases with give the following:
From this, and . is the view angle of the intruder from the own aircraft if and can be obtained from (for details, see [23]). So finally, the crossing of horizontal plane will be along the line characterized by in this direction or in the opposite direction.
Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here result as , , , or , and . This gives the coordinates of points and as , , and , , from (11) and (12), respectively. Considering the interpolation, the only singular case is where the horizontal and vertical crossing coincides and this gives immediately the 3D minimum. Otherwise, there is no problem in calculating the minimizing parameter with , , and .
2.1.3. Case 3 ()
In this case, (16) further simplifies and the LS optimal line fits will give , by fitting the second line on a constant zero parameter.
This leads to which gives and so . The first equation in (7) simplifies to giving again which gives and . Considering the only special vertical case with and , no. 1 and no. 3 cases with , give the following:
This gives as in Case 1 and which is either or radian giving in-front (of the own A/C) or behind crossing.
Regarding the calculation of the 3D minimum distance with interpolation, the final parameters here have the same result as in Case 1, , , , or , and . From this point, the same deduction can be done.
2.1.4. Summary about Special Cases
All of these derivations show that original formulae (4), (5), (7), (10), and (13) can be well applied in these special cases without any modification. This is underlined by the Monte-Carlo test run in [24] where Case 2 and Case 3 scenarios are included both with special and crossing in-front and behind vertical setups and there were no singularity or any problem with the formulae. Of course, for real data, the theoretical values will be corrupted by noise but considering multiple measurements will smoothen this out in some extent.
After developing a method to determine the 3D minimum distance (relative to intruder size) between two A/C in an encounter scenario, it should be tested in detail. Reference [24] presented promising results of a SIL Monte-Carlo simulation campaign. The current article presents results based on real flight data. This will be done in Section 5 after introducing the test A/C and scenarios and the used camera system and image processing methods.
3. Test Aircraft and Scenarios
Real flight tests are conducted with two UAVs on an airfield near Budapest (Hungary). The own A/C is a large 3.5 m wingspan, about 10-12 kg twin-engine A/C called Sindy (see Figure 6) developed in our institute (SZTAKI) to carry large payloads such as the sense and avoid system (for details and building instructions (as it is an open source project), see [27]).
The intruder A/C is a small 1.27 m wingspan 1.5-2 kg single-engine A/C which is the E-flite Ultra Stick 25e (see Figure 7). The small size of the intruder makes detection and decision about the collision difficult as the numerical results will show. However, as the S&A system works acceptably in this scenario which is close to the worst case, the system will perform even better as the conditions get better.
Both A/C are equipped with and an air pressure sensor and an MPC5200 onboard microcontroller with attitude estimation and autopilot algorithms. Waypoint or trajectory tracking (constructed from straight lines and arcs) is also possible with them. For the details of the sense and avoid hardware/software system on Sindy, see Section 4.
The S&A test flights were conducted in October/November 2016 by flying the two A/C on parallel, horizontal, straight flight paths with 15 m (close flight) and 30 m (far flight) horizontal side distances between them. The flying on nonparallel paths would be much more complicated as [28] points out. The own A/C air relative velocity was set to be 20 m/s while that of the intruder was 17 m/s giving 37 m/s closing speed in the ideal case. During the first set of test flights, the avoidance was deactivated to make in detail offline algorithm evaluation possible. Note that if an avoidance maneuver is done, the own A/C trajectory is modified and it is impossible to evaluate the closest point as if there was no avoidance. In the second set of test flights, avoidance in the horizontal plane (only considering parameters) was activated and maximum bank turn avoidance maneuvers were done when required. Setup of a safety altitude separation (15 m for close test and 10 m for far test) was mandatory in the first set and advised in the second (that is why only horizontal avoidance was applied). Depending on the initial conditions, actual wind disturbance, and turbulence, the tracking of the straight line can show some oscillatory behaviour which can compromise camera data and the precision of the S&A algorithm. A close test case and a far test case with almost straight trajectories are shown in Figure 8 while oscillating ones are shown in Figure 9.
4. Camera System
Figure 10 presents the camera system hardware and the main characteristics of the components. The system includes two Basler dcA 1280 cameras which have a 65.4° field of view (FOV). They are setup with camera angles which gives finally ±62.7° total FOV. This is in good agreement with the ±60° angle range suggested for pilots to scan for intruders in [25]. The NVIDIA Jetson TK1 board is responsible for image processing and decision making. This hardware is capable of processing and saving the 2x HD monochrome video input at about 8 fps. Communication between the autopilot and camera system is minimal. The autopilot sends the orientation of the aircraft while the camera system sends only an evasion alert and suggested avoidance direction.
4.1. Image Processing
Recent papers on flying object detection and tracking utilize deep neural networks [29]. The accuracy of these methods is impressive; however, these techniques have too large computational complexity for onboard real-time computation. The evolution of new hardware and research on more efficient neural network design will lead to robust flying object detection in the future; however, now, different methods were applied. Here, we introduce our approach which was used in the real flight tests. The task of detecting flying objects in images is easier being against the sky than against the ground. All objects which are not a cloud or the sun and have no connection to the ground are flying objects and can be determined on a single frame. Against the ground, we could detect and track objects which are moving compared to the background considering multiple frames for the detection. Our method now focused on detection against sky background with the following three stages: sky-ground separation, blob detection against sky background, and false object filtering.
4.1.1. Sky-Ground Separation
We perform sky-ground separation by horizon detection. The basic method was published in [30]. The main idea is to calculate a horizon line based on the orientation of the aircraft and the known relative orientation of the camera. This horizon line is not accurate enough, because of the small airframe deformations during maneuvers and calibration errors, but this makes it very easy to create a good sky-ground separation (Figure 11).
4.1.2. Blob Detection against Sky Background
Algorithm 1 is an example preprocessing method to extract candidate objects. Candidate objects include real flying objects with false positives (cloud edges and so forth). A successfully detected aircraft and its enlarged and shaded region can be seen in Figure 12. Thus, an object filter is applied to classify flying objects.
| ||||||||||
Algorithm 1. Preprocessing. | ||||||||||
4.1.3. Object Filter
Most of the false sky objects can be eliminated efficiently with simple filtering rules. The main rule defines threshold on the pixel variance in the neighborhood of the object mask which eliminates cloud edges. These handcrafted rules were enough for successful flight tests in good conditions (homogeneous clear or cloudy background); however, in some cases, we had still too many false positive objects which can cause false evasions. With the next-generation faster NVIDIA TX1-based camera system, classification of candidate objects could be performed with a small convolutional neural network.
4.1.4. Parameters to the Collision Decision
Finally, the horizontal and vertical centroid positions () and object sizes () are obtained after detection. From these parameters, a rectangle is created which is egomotion compensated (see [10]). This means rotation from the camera to the body, from the body to the trajectory, and then from the trajectory to the camera coordinate system. The trajectory system is a body system rotated to be parallel to the straight flight trajectory. This way, the intruder direction is evaluated relative to the trajectory direction irrespective of the current orientation of the own aircraft. After egomotion compensation, () and () are recalculated and then passed to the algorithm in Section 2.
5. Real Flight Test Results
From the first set of flight tests (without avoidance), several important parameters can be evaluated such as the detection distance of the intruder, the time frame to make a decision and execute avoidance (from time of detection to time of CPA), and the fps (frames per second) rate of the camera system. 8 close and 9 far encounters were flown, and data synchronization between the two A/C was done considering the global GPS UTC (coordinated universal time) time stamps. Note that the GPS frequency is 5 Hz so the time resolution of data is 0.2 s. The minimum, maximum, and mean values and the standard deviations (STD) of the above parameters are summarized in Table 1 for the close and far encounters.
Considering the data in the table, the of the camera system with onboard image processing and decision making is between 6.3 and 8 Hz with a mean of about 7.5 Hz. This is about half or less than the 15-20 Hz in [12], but in that paper, they did ground-based calculations with a laptop computer while we applied an onboard system carriable by the Sindy A/C and making the S&A system autonomous. Future hardware developments can easily increase this rate.
The detection distance is between 122 and 204 m in the close scenarios while it is between 55 and 244 m in the far ones. Here, detection means that the tracking of the intruder by the camera system is stable and continuous from that point. The mean values are 172 and 164 m, respectively. Regarding the means, these results outperform the 60-70 m detection distance of a 1.6 m wingspan A/C in [12] (note that our intruder wingspan is only 1.27 m). The very small (55 m) detection distance in the far scenarios is an outlier (the mean 164 m is close to the close scenario 172 m mean), and we could not figure out what caused it. The flight trajectories were good, and so, the intruder could not move out of the camera system FOV so possibly some image blur or distortion could cause it.
The time frame to make a decision is 3.47-5.59 s for close and 1.77 6.35 s for far scenarios because of the same reasons as for . The means are 4.7 s and 4.5 s, respectively, which do not provide too much time to make a decision and execute avoidance. Considering the later selected 3 s TTCPA threshold which provides 2 to 4 s real time until the closest point and the required 2-3 s convergence time of our algorithm, it would be better to have 6-7 s from observation until the closest point. That is why such a small A/C as an Ultrastick aircraft is a critical case which makes avoidance with large safety distance impossible. On the contrary, our article [31] points out that for general aviation A/C (7 m and above wingspan), the time from detection to the closest point is at least 7 seconds which can provide enough time for successful avoidance. It is also worth noting that approximately 10 s [25] and 12.5 s [26] are the required time for a human pilot to detect, identify, and avoid an intruder which is outperformed by the 6-7 s time of our system. From and , the average closing speed of the A/C can be calculated which results between 31 and 40 m/s with a mean of about 36.5 m/s close to the theoretical 37 m/s value.
5.1. More Details about Intruder Detectability
An exhaustive study about the detectability of intruders with different camera systems, sizes, and speed is published in [31]. Some of the observations deduced there are repeated here to put our previous observations into context. At first, it is worth to mention that the detectability of an object by a camera mainly depends on the sensor area occupied by the image on the sensor. This area can be directly related to the object pixels through the so-called pixel pitch parameter. Another possibility can be to use the subtended angle of the object, but this suffers from distortion by approaching the edges of the image plane. So finally, the covered area (size) was considered in [31]. Table 2 repeats the considered camera parameters from there.
Our camera in the flight tests is the Basler dcA 1280. Considering the size of the Ultrastick aircraft and the average detection distance makes it possible to calculate the average detectable size in pixels as . Considering the pixel pitch, this gives about 30 μm size on the sensor. The MIN sizes for the other cameras in the table were calculated from this value. The minimum and maximum detection distances for several aircraft sizes were calculated considering the worst (dcA 1280) and best (acA 2040) cameras as shown in Table 3.
For the Ultrastick A/C, the worst distance (159 m) is close to our mean detection distance (164 m) which is not surprising as we calculated the parameters from this. Our maximum detection distance (244 m) is very close to the best result (245 m) which means that even the worst camera can give superior results in some cases but, of course, using higher-resolution cameras can improve the results. These tests in [31] and the pinhole projection formula show that the camera should have a large focal length () and small detectable occupied sensor area . That is why acA 2040 outperformed acA 2500 because is much larger and the focal length is almost the same in the latter case.
The time to the closest point of approach at the time of A/C detection strongly depends on the speed of the own and intruder A/C. A thorough evaluation of this problem is done in [31].
5.2. Results and Thresholding with Flight Test Data
The first set of flights (without avoidance) makes it possible to compare S&A system estimates with real physical data. Real 2D (only horizontal) and 3D closest points were calculated from GPS data for all close and far encounters. Then the calculated distances were normalized by intruder wingspan and height (the real height/size ratio thus results as ) to finally get . As the A/C fly on horizontal paths, there is no crossing in the vertical plane and so the vertical and the 2D horizontal CPA is the same as the that of the 3D. So, it is enough to compare the estimated and real 3D results.
The S&A algorithm was run offline for all close and far encounters, and the estimates are compared to the A/C trajectory-based real values in Figures 13–19. The LS line fits were done on 7 data points to consider about 1 s data (fps is about 7.5).
Figure 13 shows the estimated on the horizontal axis and the error on the vertical axis. Note that changes with in every time step as the intruder approaches the own A/C. These time series values are plotted together with their instantaneous errors in the figure. If the error is positive, there is more time to CPA than estimated. This representation is advantageous to consider the possibilities for decision threshold (THS) selection. The figure shows that first (when the estimated is large), the TTCPA is overestimated and then, as the intruder approaches, it is underestimated. Table 4 summarizes the observations.
According to the table, an estimated THS of 3 s will result in a decision between 2 and 4 s real time before CPA. Selecting a larger THS will not increase the possible smallest real value (2 s) while smaller THS will decrease it, so 3 s is the ideal selection.
Figure 14 shows the estimated on the horizontal axis and the error on the vertical axis. Here, the real is a constant given value in every encounter but it changes encounter by encounter depending on the shape of the flight trajectories. At the same time, the estimated one changes every time step as new and new images are considered in the estimation method. If the error is positive, the intruder is farther than estimated. It is interesting to see that for close encounters, is usually underestimated, while for far encounters, it is usually overestimated. Table 5 summarizes the observations. According to the table, the avoidance of an intruder with a horizontal CPA of 4 or below can be guaranteed by any THS and only intruders above 25-28 CPA can be surely considered as nonthreatening. Avoidance of any intruder at or below 4 CPA will not give well clear but can provide the last chance to solve the situation without collision.
In case of CPA THS selection, the relation of CPA estimates with the TTCPA estimates is vital to evaluate decidability (between collision and noncollision). This is because collision decision is done examining the absolute CPA estimate when estimated TTCPA is below the selected THS. That is why TTCPA-CPA nomograms should be plotted and applied for THS selection. Figure 15 shows the estimated on the horizontal axis (plotted only below 3.5 s as its THS is 3 s) and on the vertical axis together with two possible horizontal CPA THS values. The first (magenta dashed line) is 20 while the second (green continuous line) is 26. The first (20) has more close scenario crosses above it and so can result in missed detection with higher probability. Below 2.25 s estimated TTCPA, all the red crosses are below the second (26) so this should guarantee detection of close threat at least below this time. Of course, the larger THS will result in more false alarms as several far scenario circles are below it.
Table 6 shows estimation error evaluation results for vertical CPA () similarly to Table 5 based on a figure similar to Figure 14 but not presented here. In this case, intruders below 4-5 vertical CPA (relative to intruder horizontal size) can be guaranteed to be avoided and those above 10-11 CPA can be considered as nonthreatening.
Figure 16 shows the estimated on the horizontal axis and on the vertical axis together with two possible vertical CPA THS values. The first (magenta dashed line) is 5 while the second (green continuous line) is 20. The first has several circles (which is the vertical close scenario with 10 m altitude separation) above it and so can result in missed detections. The second has only a few circles above it, but all the crosses are below if . This means that on one side, it possibly will not do missed detection but on the other side, it will generate false alarms for all far vertical encounters.
Figure 17 shows the real on the horizontal axis and the error on the vertical axis. The figure shows that the direction estimation is very uncertain so it is not advisable to use it for precise avoidance. This large uncertainty is caused by the oscillating flight paths despite the egomotion compensation of image data. Note that egomotion compensation removes only the rotational component of the oscillation; the position change disturbances are not removed. Detailed examination of the estimates shows that their sign is correct and so at least the side of the intruder can be decided based on them.
Figure 18 shows the real on the horizontal axis and the size ratio error on the vertical axis. Here, 0.26 is the real size ratio. The figure shows that is overestimated most of the time.
Figure 19 shows the nomogram with thresholds from Figure 16 transformed by dividing by 0.26 which is the real size ratio. The figure shows that here again, thresholds which cause missed detections or false alarms can only be selected as the vertically close and far cases are overlapping even for small estimated TTCPA.
Finally, a decision test campaign was run offline on the real flight data applying five THS pairs shown in Table 7. The decision is done if the estimated is below its threshold. Collision is decided if the estimated CPA is below the given THS (this makes possible a more precise decision than in [12] where every intruder is avoided which is in the image for a longer time). Results are summarized in Table 8 for the close encounters and Table 9 for the far encounters referencing the THS case numbers from the first row of Table 7. In the close encounters the horizontal decision should be collision and the vertical non-collision (NC). In the far encounters horizontal non-collision and vertical collision decisions are expected. In Tables 8 and 9, the numbers mean the real TTCPA time in seconds at the time of collision decision if collision is decided. In case of NC decision, an NC symbol is used.
Table 8 shows that no. 1 to no.3 THS pairs well detect the possibility of collision in the horizontal plane. The only difference is at the time of detection.
The earliest detection is achieved by the and pair (no. 1) while the latest by the and (no. 3). The real TTCPA values are between 2 and 4 s in most of the cases as predicted in Table 4. The only outlier is the first flight where the estimated TTCPA crosses the THS late after a sudden decrease. In the vertical (nos. 4 and 5) plane, NCs are expected while 50% false alarm is achieved by no. 4 and 100% by no. 5. In view of the 8 − 10 range of real values based on GPS data, this is not surprising as only in above 10-11 is a guarantee of NC decision present both with 5 and with 20 THS (see Table 6).
Table 9 shows that no. 1 to no. 3 THS pairs generate a lot of false alarms as follows: 8/9 (89%), 6/9 (67%), and 4/9 (44%). Given the 18−30 range with mean 25 of real values based on GPS data, this is not surprising as only in above 25-28 is a guarantee of NC decision present both with 20 and with 26 THS (see Table 5). In the vertical plane, there is no missed detection; the only difference between , (no. 4) and , (no. 5) is the time of decision. The latter decides earlier on which can be advantageous to execute avoidance.
Summarizing the results, the no. 2 THS pair ( and ) can be useful in the horizontal plane as it provides more time for avoidance than the no. 3 pair though it gives very high false alarm rate. In the vertical plane, the no. 5 pair ( and ) is better as it provides more time for avoidance despite the 100% false alarm rate. Note that the false alarm rates should get lower as the horizontal or vertical distance is increased between the A/C.
A second set of flight tests was done in 2016 (on a different day) by applying the and (no. 3) THS pair only in the horizontal plane and activating the S&A avoidance. In the close scenarios, a 0/8 (0%) missed detection rate was observed while in the far ones, the false alarm rate was 4/8 (50%). All these rates are close to the ones estimated from the tables. Checking the A/C trajectories with the avoidance has shown that the collision decision was made sometimes too late to significantly increase the distance between A/C. This could be predicted from the detection time results in Table 8. Possibly, the 3 s TTCPA THS can improve these results; this requires a future test campaign. Ground and aerial (onboard) videos of the second set of flight tests in 2016 can be found in the supplementary material and on our Youtube channel called AeroGNC.
6. Conclusions
This paper presented the real flight validation of 3D collision decision based on monocular images in aircraft sense and avoid. After the review of literature and author’s previous works, the theoretical basics were briefly summarized targeting the estimation of 3D time to the closest point of approach and closest point of approach. The feasibility of possible special cases with the derived formulae is examined, and there is no need to handle any of them separately—the original formulae can handle everything well. Then the test UAVs and flight test scenarios were introduced together with the onboard camera and computing system. A short analysis of intruder aircraft detectability with different camera systems is provided referencing the author’s separate exhaustive work about this topic. Real test flights with close and far encounters without avoidance were conducted, and the estimated collision parameters were compared to the aircraft trajectory-based real ones. A strategy to select the decision thresholds was proposed, and then the offline decision results were presented and evaluated and finally compared to real flight test results (with avoidance) with a given threshold pair. The final conclusion is that the proposed system is capable of detecting and attempting to avoid close intruder aircraft even if it is as small as 1.27 m (wingspan) but of course larger size intruders are easier to be detected and avoided. Intruders with 7 m wingspan and above are predicted to be avoided with enough safety distance, but this also depends on own aircraft maneuvering capabilities and intruder speed. By applying other camera systems, this can be further improved.
Future research should focus on two different topics. First, a relation between software-in-the-loop simulation results in previous works of the authors [23, 24] and the real flight tests should be established to make it possible to run close to real simulations with virtual intruders. The final goal is to tune the decision thresholds based solely on the knowledge of own aircraft dynamics, camera system parameters, and possible intruder size and velocity ranges all integrated in a realistic simulation environment. Second, an appropriate avoidance strategy based on the dynamic capabilities of the own aircraft and the estimated collision parameters should be proposed and tested.
Data Availability
The numerical data used to support the findings of this study is included within the article. The video recordings used to support the findings of this study are included within the supplementary information file(s) and are available on Youtube AeroGNC channel.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this article.
Acknowledgments
This work was supported by the Institute for Computer Science and Control, Hungarian Academy of Sciences (MTA SZTAKI) (Grant number 008/2016). This paper was supported by the Janos Bolyai Research Scholarship of the Hungarian Academy of Sciences. The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 690811 and the New Energy and Industrial Technology Development Organization, Japan, under grant agreement no. 062600 as a part of the EU/Japan joint research project entitled “Validation of Integrated Safety-enhanced Intelligent flight cONtrol (VISION)”.
Supplementary Materials
Supplementary 1. SAA_decision.mp4: successful small (1.27 m wingspan) Ultrastick UAV detection based on the monocular camera system. Image processing and decision about the possibility of collision are done onboard the camera-carrying Sindy UAV (3.5 m wingspan). The date of flight is in October 2016. Successful intruder detection and decision about the threat in a near scenario (15 m horizontal distance between aircrafts) can be seen. This is a tuning flight, avoidance is not activated, and altitude difference (15 m) guarantees safety of the aircraft. This video is related to the first set of flight tests and the selection of thresholds.
Supplementary 2. SAA_success.mp4: successful small (1.27 m wingspan) Ultrastick UAV detection and avoidance based on the monocular camera system. Image processing, decision about the possibility of collision, and avoidance maneuver design are done onboard the camera-carrying Sindy UAV (3.5 m wingspan). The date of flight is in November 2016. Successful intruder detection, decision about the threat, and avoidance in a near scenario (15 m horizontal distance between aircrafts) can be seen. This video is related to the second set of flight tests executed with the and (no. 3) threshold combination.
Supplementary 3. SAA_success_from_ground.mp4: successful small (1.27 m wingspan) Ultrastick UAV detection and avoidance based on the monocular camera system. Image processing, decision about the possibility of collision, and avoidance maneuver design are done onboard the camera-carrying Sindy UAV (3.5 m wingspan). The date of flight is in November 2016. Successful intruder detection, decision about the threat and avoidance in near scenarios (15 m horizontal distance between aircrafts), and successful decision about the safe distance in the far scenario (30 m horizontal distance between aircrafts) can be seen. This video is related to the second set of flight tests executed with the and (no. 3) threshold combination and shows the ground recording of the avoidance case in SAA_success.mp4 also. First, the camera-carrying aircraft is shown, second and third two near scenarios are shown where avoidance is executed, and fourth, a far scenario is shown where there is no need for avoidance. Note that the aircraft seem to be very far from each other in all of the scenarios because of the safety altitude separation.