A Low-Cost and Ultralight Unmanned Aerial Vehicle-Borne Multicamera Imaging System Based on Smartphones

Yang, Huachao; Bian, Hefang; Li, Bin; Bi, Weihua; Zhao, Xingtao

doi:https://doi.org/10.1155/2022/8524400

Mathematical Problems in Engineering

On this page

Abstract Introduction Conclusions Data Availability Conflicts of Interest Authors’ Contributions Acknowledgments References Copyright Related Articles

Special Issue

Mathematical Problems in Geomatic Spatial Information Technologies

View this Special Issue

Research Article | Open Access

Volume 2022 | Article ID 8524400 | https://doi.org/10.1155/2022/8524400

A Low-Cost and Ultralight Unmanned Aerial Vehicle-Borne Multicamera Imaging System Based on Smartphones

Huachao Yang,¹Hefang Bian,¹Bin Li,¹Weihua Bi,²and Xingtao Zhao³

Academic Editor: Adnan Maqsood

Received12 Aug 2021

Revised28 Nov 2021

Accepted17 Dec 2021

Published13 Jan 2022

Abstract

Newly developed oblique photogrammetry (OP) techniques based on unmanned aerial vehicles (UAVs) equipped with multicamera imaging systems are widely used in many fields. Smartphones cost less than the cameras commonly used in the existing UAV OP system, providing high-resolution images from a built-in imaging sensor. In this paper, we design and implement a novel low-cost and ultralight UAV OP system based on smartphones. Firstly, five digital cameras and their accessories detached from the smartphones are then fitted into a very small device to synchronously shoot images at five different perspective angles. An independent automatic capture control system is also developed to realize this function. The proposed smartphone-based multicamera imaging system is then mounted on a modified version of an existing lightweight UAV platform to form a UAV OP system. Three typical application examples are then considered to evaluate the performance of this system through practical experiments. Our results indicate that both horizontal and vertical location accuracy of the generated 3D models in all three test applications achieve centimeter-level accuracy with respect to different ground sampling distances (GSDs) of 1.2 cm, 2.3 cm, and 3.1 cm. The accuracy of the two types of vector maps derived from the corresponding 3D models also meet the requirements set by the surveying and mapping standards. The textural quality reflected by the 3D models and digital ortho maps (DOMs) are also distinguishable and clearly represent the actual color of different ground objects. Our experimental results confirm the quality and accuracy of our system. Although flight efficiency and the accuracy of our designed UAV OP system are lower than that of the commercial versions, it provides several unique features including very low-cost, ultralightweight, and significantly easier operation and maintenance.

1. Introduction

Low operation and hardware costs of the low altitude unmanned aerial vehicles (UAVs) compared to the areal imaging with high-quality sensors from manned airborne platforms make UAVs an important means for aerial photogrammetry in many applications areas [1–4]. Unmanned aerial system (UAS) provides competitive advantages especially for local area applications and in cases where repetitive data collection or rapid response is required. Emerging field areas involve in various monitoring applications, for example, land cover or vegetation monitoring [5–7], geometric changes of the environment such as earthquake disasters [8], landslide [9], volumetric changes of open-pits [10], and archaeological sites [11, 12]. UAS is also used as a modern technique to obtain digital linear graph (DLG), digital ortho map (DOM) that provides useful field metrics, essential digital elevation model (DEM), and digital surface model (DSM) [13, 14]. Four geospatial data, that is, DLG, DOM, DEM, and DSM provided by the UAS, also referred to as 4D products in the digital photogrammetry field, are essential in spatial analysis and geographic information system (GIS) applications [13, 14].

The development of multicamera airborne imaging systems with large off-nadir angles facilitates obtaining denser and more complete 3D photogrammetric point clouds through multiview stereo dense matching [15, 16]. In general, an oblique photogrammetric (OP) equipped with multicamera airborne imaging system is able to acquire vertical and four oblique aerial image perspectives for a strip. Oblique aerial imagery is acquired with a tilt-viewing angle at approximately 45° and provides both top and lateral information for the ground objects. The selected viewing angle provides the user with maximum facade information while minimizing the occlusion and geometrical deformation effects [17–22]. In contrast with the VP, OP is beneficial for capturing spectral and geometrical facade information with less perspective distortion.

The quality and accuracy of the 4D products obtained by the OP are greatly higher than that of VP. The results are centimeter-level resolution and accuracy products, especially for precise 3D model reconstruction. Such a precise 3D model is also used in the automatic generation of 3D photorealistic city mesh models. It allows the general public to intuitively assess a 3D urban environment without a need for training. Although multicamera imaging systems have distinctive merits as discussed above, such systems are often too heavy to be mounted on UAV platforms with limited payload capacity. For instance, some large format multicamera systems (e.g., the Microsoft/UltraCam Osprey, Hexagon/Leica RCD30, IGI Penta-DigiCam, and Track-Air MIDAS) must be mounted on manned aerial vehicles due to their weight. This, in addition to the high costs, makes using such equipment inaccessible not only to common users but also to many companies. To use such a technology on UAV platforms with limited payload capacity, multicamera systems with small and lightweight sensors are needed.

Advancements in computer and information technologies have led to many applications affecting our day-to-day activities. A prominent example of such technologies is the new generation of smartphones. A smartphone is a mobile communication terminal equipped with an open-type operating system. The emergence of mobile technology has affected technological aspects of life, as well as users’ expectations and perceptions. A variety of MEMS (micro-electro-mechanical systems) sensors and other devices are also incorporated into smartphones to improve the user experience. One of the most attractive smartphone features is the integrated CMOS (complementary metal-oxide semiconductor) camera. Many researchers investigate the photogrammetric applications of smartphone cameras. For instance, Micheletti et al. [23] investigated the potential of a straightforward structure-from-motion (SfM) approach using basic smartphone imaging technology and partial and fully automated data-processing resources, and two cased studies have been chosen for the analysis. Tavania et al. [24] proposed a workflow that utilizes consumer-grade smartphone sensors to produce fully georeferenced SfM surface reconstructions of large-scale (i.e., a few hundred meters to several kilometers wide) scenes; the information provided by GNSS, accelerate/gyroscope, and magnetometer allows registering 3D surface. Shults et al. [25] employ HTC Desire and Meizu M3 Max smartphones to model a cultural heritage site. Kim et al. [26] evaluate the feasibility of employing a single lightweight smartphone as the payload for photogrammetric applications.

In this paper, we present a novel low-cost and ultralight UAV OP system based on smartphones. The cost of a smartphone is usually less than commonly used cameras in the existing UAV OP system. Our proposed system is a smartphone-based multicamera (SMC) system in which five digital cameras detached from five smartphones are rearranged and fitted into a very small device to synchronously shoot images at five different perspective angles. We then design an independent module for automatic capture control. Our implemented SMC system is then installed on the modified version of an existing lightweight UAV platform. The results of our proposed system are then compared with other commercial UAV OP systems through a field test with three typical applications. The proposed system is also fabricated to facilitate, accelerate, and ensure a cost-effective solution for common users.

2. System Description

2.1. Overview

The proposed smartphone-based UAV OP system comprises two main parts (see Figure 1): the flight platform and the SMC. The flight platform system includes a rotary wing quad-rotor UAV, a flight control system, and a ground flight-visualization monitoring subsystem. The SMC system includes a multisensor imaging module, a multimainboard module with a data storage function, an automatic image-taking control module with synchronous positioning and data storage ability, and an electric power module.

2.2. Selection of the UAV Platform System

The UAV platform system in our research is a DJI Phantom 4 Pro (P4P) manufactured by Da-Jiang Innovations Science and Technology Co. This UAV consists of four parts: a four-rotor UAV platform, CMOS camera with three-axis (pitch, roll, and yaw) stabilization gimbal, flight control system, and ground flight-visualization monitoring system (see Figure 1). The flight duration of the system is approximately 30 min, and the maximum flight speed is 20 m/s, with a 6,000 mAh 4S battery. The DJI flight control system is adopted with the latest DJI navigation control algorithm, and the new redundant design of dual IMU (inertial measurement unit) is used for real-time data backup. P4P is equipped with a 20 mm (35 mm equivalent) lens and 1 in sensor capable of producing 20 megapixels (MP) images. The controllable range of the pitch of the stabilization gimbal is from −90° to +30°, which enables the camera to shoot images from all possible angles. The ground flight-visualization monitoring system uses an Apple iPad with a touch screen. P4P is selected in our study due to its low weight, relatively low cost, and efficient electric power system. We also remove the camera and its accessories mounted under P4P and substitute them with our SMC system.

2.3. Description of SMC System

2.3.1. Selection of the Smartphone

Various types of CMOS sensors are built into smartphones to enable them to take high-resolution images. Here, we use Nokia N 808 PureView, released in 2012, to modify its imaging sensor. This smartphone has a 1.3-GHz ARM11 processor, 512 MB of internal RAM, and 16 GB of available user storage. The camera in this smartphone combines a longer focal length with a large aperture that enables shooting at closer distances and creating wide-ranging depth of field. The CMOS sensor size (1/1.2 in) in this smartphone is relatively larger than that of the others. Nokia PureView Pro imaging technology in this smartphone is a combination of a large, super-high-resolution 41 MP, and high-performance Carl Zeiss optics. The large sensor enables pixel oversampling, that is, combining many pixels into one perfect pixel. PureView imaging provides higher image quality, lossless zoom, and superior low light performance. These advantages are the reasons behind choosing this smartphone. Nokia N 808 PureView smartphone provides two imaging modes with the same lens but with different image resolutions and pixel sizes (for detailed specifications, see Table 1).

2.3.2. Multisensor Imaging Module

The multicamera imaging system developed in our research includes one vertical lens and four perspective lenses. This setting enables us to acquire one vertical and four oblique aerial images at an exposure point. We use five smartphones; however, it is not possible to simply fix them on the UAV due to the limited space and payload provided by the P4P UAV platform. Therefore, some modifications are made, and we detach the cameras and rebuild them into an integrated SMC system. This significantly reduces the weight of the payload. We disassembled the smartphones and discarded the unnecessary components (such as mobile shells, iron screws, etc.), and only the indispensable parts involving in image-taking functions are retained. Figure 2 exhibits this setting, where the main two parts of a smartphone mainboard and a CMOS sensor are connected using a flexible flat cable.

The isolated five cameras are then rearranged to fit into a small compact space. Figure 3 shows the spatial configuration, where the pink dots represent the camera lens. As it is seen, the cameras are installed on the bottom shell surface (this surface is also parallel to the plane of the mainboard). One camera is set on the center of the lens cluster, and the other four are symmetrically distributed around the center and form a square with a side length of about 50 mm (see the white dot frame in Figure 3).

(a)

(b)

As for the observation views of these cameras, the optical axis of the central lens is orthogonal to the bottom shell surface, and the slant angle of observation view between them is set to 90°. The corresponding angle of the other four lenses is set to about 30° to 45° in order to ensure imaging quality, as well as the same ground sampling distance (GSD) for all cameras. In our study, this value is set to 30°. Figure 4 shows the layout of five images with different angles captured by the SMC system at an exposure station, where the red frames highlight the corresponding objects in this image group.

Compared with the professional photogrammetric cameras, the smartphone camera has relatively lower image quality; however, Nokia N 808 PureView has a better performance than that of other available smartphones. In order to further ensure the captured image quality, we developed several following measures:(1)Adding filters to the lens to avoid overexposure: A common artifact in photographs is overexposure due to the bright scene features exceeding the abilities of the camera. This results in image areas appearing flat and lacking detail. To address this issue, a feasible solution is to attach a neutral density (ND) filter to the camera lens. In photography and optics, an ND filter is a filter that reduces or modifies the intensity of wavelengths or colors of light equally, without changing the hue of color rendition. Practical experiments also confirm that attaching an ND filter is effective in reducing the overexposure phenomenon. Figure 5 shows the comparative results before and after using an ND filter.(2)Disabling the autofocus and antishake functions: Generally, the image data used for photogrammetric applications are obtained by a camera with a fixed focal length; otherwise, the interior orientation parameters (IOPs) of the lens may be changed. This reduces the accuracy of the subsequent image processes, such as tie points extraction and matching, image triangulation, and so on. A consumer-grade mobile phone camera usually takes images using an autofocus mode and adopts an image stabilizing system that enables users to take better pictures. These however lead to slight changes in the focal length that are not preferred in many photogrammetric applications. For this reason, we disable the autofocus shooting mode and antishake function through the operating system of the smartphone before disassembling it.(3)Optimization of the imaging mode: Table 1 shows two commonly used imaging modes of Nokia N 808 PureView. As it is seen, the two imaging modes have the same lens parameters and CMOS optical format; however, they have different image resolutions of 41 MP and 12 MP, and their corresponding pixel sizes are 1.4 μm and 2.6 μm, respectively. The higher the image resolution, the better the image is to the photogrammetric applications. For practical applications and 3D model reconstruction, however, very small pixel size is in fact not suitable, as it usually leads to “distortion” in the 3D model due to pixel oversampling. A typical example is shown in Figure 6, where it is clearly seen that the 3D reconstruction results produced by Mode 1 with a pixel size of 1.4 μm demonstrate a serious “twisted” phenomenon within the area marked by the red ellipse, and the corresponding 3D model is poorer compared to that of Mode 2 with a pixel size of 2.6 μm. Therefore, we use Mode 2 in our research. Another advantage of selecting Mode 2 is its smaller required storage capacity. In other words, for the same flight mission, the amount of needed memory required for Mode 1 camera operation is more than three times larger than that of Mode 2. Therefore, adopting Mode 2 greatly reduces the data processing and storage requirements of the photogrammetric.

(a)

(b)

(a)

(b)

2.3.3. Multimainboard Module

As discussed in Section 2.3.2, apart from the cameras, five independent mobile mainboards were also obtained after segregating them from a corresponding main body of a smartphone. Similar to lenses, mainboards are also rearranged in order to fit the limited space of the integrated SMC system. As shown in Figures 1 and 3(b), these five mainboards are closely superimposed on each other, forming a cuboid. This layout has a small occupied space and is also convenient for packages.

We further reduce the weight by pruning the original smartphone mainboard and only keeping the necessary parts related to image-capturing and data storage functions. Each mainboard is linked to the corresponding lens by a flexible flat cable (see Figure 2). The main function of this module is to receive control instructions sent from the automatic capture control module and to transmit them to the multisensor imaging module, which also stores the image data.

2.3.4. Automatic Capture Control Module

In practice, no extra ports as the one used for the flight control system are built into commercial light UAVs. Therefore, automatic image-taking control of the SMC system cannot be implemented only by using the existing flight control system. Therefore, an independent module of the flight control system is used to automatically take images based on Pixhawk open-source flight control system. Pixhawk is considered as the UAV equivalent of “Android” for smartphones [26–28], and its supported PX 4 firmware provides a message mechanism that allows multiple processes to open the same device file and exchange and share data. The developed flight control system mainly includes three components including global positioning system (GPS), IMU, and image-taking control units and implemented on a PX 4 board, as shown in Figure 1.

In addition to the digital camera, there are several other built-in MEMS sensors in smartphones for various applications. Theoretically, we can obtain the 3D position from the smartphone GPS and the attitude data from various sensors (magnetometer, gyroscope, accelerometer, and others). Since, in our system, smartphone cameras are mounted at the bottom of the UAV body; the reception function of the GPS unit might be unstable due to signal blockage. The attitude MEMS sensors are low-cost, compact, and power-efficient; however, they may result in large attitude drift especially where the UAV is moving at a relatively high speed and in long distances [29, 30].

By considering the above factors, the system is designed to collect position and attitude information via an external GPS receiver and an independent IMU unit (see Figure 1). Localization of the multisensory array is obtained using the onboard UAV GPS and linked to an external antenna connector provided by Pixhawk. Pixhawk flight control system is equipped with its own IMU unit with a three-axis gyroscope, three-axis magnetometer, and three-axis accelerometer, from which we can detect the orientation of the device. Measured data values are then stored in an SD card or internal storage of PX 4. The image-taking control unit plays an important role in the SMC system. The camera trigger driver provided by PX 4 allows the use of onboard AUX ports to send out pulse width modulation (PWM) signals in order to trigger a camera. This can be used for multiple applications including time-stamping photos for aerial surveying and reconstruction, synchronizing a multicamera system, or visual-inertial navigation. To transmit a pulse, a MAVLink message is published containing a sequence number (e.g., the image sequence number of the current session) and the corresponding time stamp. Four different modes are supported and controlled by the TRIG_MODE parameter. In our study, a camera trigger with a certain time interval is set. This allows the multicamera system to shoot images at a regular time interval and further makes a slight difference of exposure time between different imaging sensors in μs timescale.

In our system, the constant parameter of the time interval is set to 3 s. Moreover, for intercommunication, we develop protocols to control the individual sensors or devices, as well as the software for microcontroller STM32F427 of PX 4. This software is developed in Arduino IDE using C++ programming language with Visual Studio 2013 platform. Once the cameras are triggered, the SMC system synchronizes the positioning and orientation system (POS) with the cameras. After receiving and recording POS information, the data is then converted into a complete data packet, and specific communication is formed. The protocol stores the POS data on internal storage in PX 4 with a capacity of 8 GB.

2.3.5. Power Module

The power module uses an external battery with a capacity of 3,000 mAh and provides suitable and stable voltage to the modules on board of the UAV. An external power port is also designed and fabricated to ensure continuous operation in the fieldwork by substituting the exhausted battery.

2.4. Integration and Workflow

Besides the external GPS antenna and external battery (Figure 7), the other parts of the SMC system are packed into a small chamber. The frame is also created by a 3D printer using lightweight acrylonitrile butadiene styrene plastic (ABS plastic). The integrated SMC system and its important dimensions are shown in Figure 3. Furthermore, the SMC system should be integrated into the UAV platform by simply substituting its original camera system. A fully integrated smartphone-based UAV OP system is shown in Figure 7. The flight time of the integrated UAV OP system powered with the 6,000 mAh 4S battery is an average of 18 min considering different weather conditions and flight circumstances.

The procedure of a flight cycle for collecting images for OP applications is carried out based on the following four steps, as also shown in Figure 8.

Step 1. Ready to take off. The flight mission is planned by the professional software installed in-ground flight-visualization monitoring system of DJI P4P. The time interval for image-taking is manually set to 3 s so that a reasonable degree of overlap is achieved according to the UAV altitude and speed corresponding to the level set prior to performing a flight mission. The flight mission is transmitted to the flight control system of P4P and the aircraft is ready to take off.

Step 2. Ready to capture images. Turn on the cameras by the camera power on/off button, as shown by a white dot in Figure 3(a). The multimainboard module is invoked and triggers a multisensor module to be ready for capturing images.

Step 3. Initiate the flight mission. In this step, we firstly open the capture function possessed by the automatic image-taking control module by capturing the power on/off button, as labeled with a yellow dot in Figure 3(a). Then, this module transmits a control command at a regular time interval of 3 s to the multisensor module. The latter begins capturing images with the same interval and synchronously record and save POS information. In other words, once the automatic image-taking control module is triggered, the camera module begins taking images, and the UAV should take off to perform the flight mission predefined in Step 1.

Step 4. Download data. After the UAV landed, a flight duration of about 20 min terminates. We firstly disable the capture function by capturing the power on/off button (marked by a yellow dot in Figure 3(a)). The images and POS data are stored in the internal memory of the mainboard module and image-taking control module, respectively. The data is then downloaded through the corresponding external ports through serial cable shown in solid green and solid yellow frames in Figure 3(a).
If the fight mission requires more than one flight circle, then repeat Step 2 to Step 4.

3. Application Examples of the System

3.1. Application Sites and Purposes

Our experiment is arranged to take place in three typical experimental sites, as shown in Figure 9, where we use the proposed smartphone-based UAV OP system for photogrammetric measurements. The areas of these three sites are about 0.06 km², 1.89 km², and 0.13 km², respectively. Both sites 1 and 2 are in the plain regions with ground elevations of approximately 34.52 m and 42.35 m, respectively. Site 1 is at the rural area, where the houses are dense, and site 2 is at the urbanized area in the city with dense and high buildings, and site 3 is at the mountain area containing large topographic variations ranging from 237.83 m to 313.80 m. The first research site is chosen to verify the feasibility of using our system for cadastral and real estate surveys. The second site involves the smart city research area, where the UAV OP system is used as an enabler to generate vivid and precise 3D models to be compared with manual 3D reconstruction methods [31]. Site 3 is an open-pit bauxite mine, and the research purposes of this site involve the use of databases obtained by the system to provide references for geological environment monitoring and the treatment measures adopted after it is abandoned [32, 33].

(a)

(b)

(c)

(d)

(e)

(f)

3.2. Data Acquisition

Measuring the ground control points (GCPs) is a crucial step necessary in the photogrammetric process and needs to be carried out before the actual flight test. For an accurate “coordinate” of GCPs, we use a “GNSS system” in real-time kinematics (RTK) mode, and the observation time at each point is about 15 seconds. This topographic and geodetic instrument measures the coordinates of the center of the GCPs with a horizontal precision between 0.014 m and 0.030 m, and vertical accuracy between 0.030 m and 0.050 m at each point. To ensure this level of accuracy, RTK corrections are carried out by using the closest national permanent GPS reference stations around each research site. In order to evaluate the real accuracy of the outputs, a certain amount of checkpoints (CPs) well distributed in the site area are measured at once, as shown in Figure 9. The ratios of GCPs/CPs for these three considered sites are 15/6, 108/24, and 8/6, respectively.

It is worth noting that for site 1 with a higher accuracy requirement of real estate survey, apart from the above-mentioned conventional CP type measured by RTK, we also measure a certain amount of house corners only with planar coordinate as the CPs by the total station (TS) with higher accuracy. In this method, the GPS control surveying is carried out first, and then the data is processed for network adjustments. According to our result of network adjustments the GPS horizontal and vertical accuracy is 6 mm, and 15 mm, respectively. Subsequently, the traverse accuracy based on the results of the GPS control survey by the TS is about 1/23,000, and the horizontal accuracy of the total station surveying is within ±10–15 mm. For research site 3 with a relatively large topographic variation, we further measure several elevation CPs by RTK to evaluate the altimetric accuracy. Therefore, the CPs used for different sites and purposes can be classified into three categories: (1) general CPs with horizontal and vertical coordinates by RTK survey for all research sites, (2) house corner CPs only with horizontal position by TS survey for research site 1, and (3) elevation CPs only with vertical location by RTK for research site 3, which are denoted for brevity by HV-CPs, H-CPs, and V-CPs, respectively. The number of measured H-CPs and V-CPs is 48, and 35, respectively, as indicated by crosses and dots in Figure 10.

(a)

(b)

For comparison of our OP system with the others, we also use a digital-camera-based five-camera imaging system mounted on the UAV platform of six-rotor DJI M600 Pro that was chosen for collecting images for each site. Detailed information about flight parameters for two UAV OP systems are listed in Table 2.

3.3. Data Processing

The 3D reconstruction is built upon structure from motion (SfM) software. Some SfM-based commercial software such as ContextCapture (CC), Street Factory, Agisoft PhotoScan, and Pix4Dmapper are widely used in various applications. Unlike traditional photogrammetry, SfM does not require IOPs, and calibration and orientation parameters are automatically calculated using a sufficient amount of GCPs and tie points identified by image matching technology [34]. In our study, CC is selected to produce the 3D models and DOMs for each research site. To obtain the topographic and the cadastral map with vector data formats, we further use MapMatrix software developed by VISIONTEK INC. to collect the vector data directly from the 3D model in a semiautomatic manner.

3.4. Results and Analysis

The right column of Figure 9 shows the DOM of each research site. The 3D models and their corresponding details for each research site are also shown in Figure 11. The real estate map of site 1 and the topographic map of site 3 that are derived from the corresponding 3D models are shown in Figure 10.

(a)

(b)

(c)

We use the root-mean-squared error (RMSE) of horizontal (RMSE_H) and vertical (RMSE_V) directions calculated from the corresponding formulation of (1) and (2) as the precision indexes to evaluate the real accuracy of the generated various data products:where n is the number of CPs, i represents the i^th CP, and , , and are the deviations between field survey data and the measured data from the digital products.

In each of these three scenarios, for three kinds of CPs, we obtained the RMSE values according to (1) and (2). The measured data of H-CPs used for research site 1 are obtained manually from the real estate map (Figure 10). With respect to the V-CPs used for site 3, the measured data are derived from a topographic map by linear interpolation of contour lines data (Figure 10). The comparative results of the two OP systems are listed in Table 3. In Figure 12, we further plot the comparison results of RMSE values for every H-CPs and V-CPs for the two UAV OP systems.

(a)

(b)

As is seen in Table 3, by taking the RMSE values of HV-CPs as indicators, the overall centimeter-level accuracy for all 3D realistic models of three research sites can be obtained for both horizontal and vertical directions. The RMSE_H values of sites 1, 2, and 3, are 3.7 cm, 6.6 cm, and 8.1 cm, respectively, with regard to the GSD values of 1.2 cm, 2.3 cm, and 3.1 cm. The corresponding RMSE_V values are 4.2 cm, 7.1 cm, and 8.9 cm, all of which indicate that the precision of the 3D model is directly proportional to the GSD of the acquired images except the amount and distribution of the GCPs used. Table 3 further indicates that both the real estate and topographic maps obtained from their corresponding 3D realistic models reach a higher precision with centimeter-level. For example, for particular applications of sites 1 and 3, the planimetric and altimetric accuracy assessed by H-CPs and V-CPs are about 4.6 cm and 9.6 cm, respectively, both of which meet the precision requirement regulated by the corresponding standards for surveying and mapping in China. Comparing the above results with the commercial UAV OP system demonstrates similar behaviors to our UAV OP system and has no significant precision improvements, although there are relatively significant differences between CPs’ detained error distributions, which can be seen from Table 2 and Figure 12.

In addition to investigating the spatial reliability of the system, we evaluate its textural quality. For this purpose, besides the 3D models, DOMs of three regions are considered (the right column in Figure 9). Figures 9 and 11 show that both the DOMs and 3D models generated from our system have richer textural quality, sharp contours, main characteristic structure details; the transition between rock stones are distinguishable and can represent the actual color of different ground objects.

As for the data acquisition efficiency, it is seen in Table 2 that although the flight duration has no significant difference, the total flight time of our OP system is substantially longer than that of the commercial OP system with the same GSD and image overlaps. The main reason is that the latter has a relatively long focal length and a large image resolution that directly leads to a larger ground coverage than our system given the same flight parameters. Table 2 indicates that the overall weight of the system including the UAV and multicamera system is 1.642 kg, and the total cost is 19,000 ¥, respectively. These are far below the corresponding values of 12.104 kg and 158,000 ¥ of the commercial UAV OP system. These suggest that although both the work efficiency and spatial accuracy of our system is not better than that of the commercial one, our proposed system provides unique features including low-cost and ultralightweight that make operating such a UAV system much easier, especially for common users with a limited technical background in this area.

4. Conclusions

This paper presented a novel low-cost and ultralight smartphone-based UAV OP system operating onboard a UAV. The system consists of a GPS/INS positioning system used to receive 3D location and attitude data and five smartphone-based multicamera imaging systems capable of acquiring high-resolution images. The system was planned for OP applications especially for realistic 3D modeling of target objects. The performance of our system was fully tested using three different application examples. For all applications conducted in our research, the achieved location accuracy of the corresponding 3D model of the system was satisfactory compared with the existing commercial OP system. The centimeter-level accuracy for both horizontal and vertical directions assessed by HV-GCPs were obtained for different GSDs of 1.2 cm, 2.3 cm, and 3.1 cm. Particularly, two kinds of vector maps were derived from their corresponding 3D models for real estate survey and topographic survey. The accuracy of the two vector maps were also obtained with centimeter-level by evaluating H-CPs and V-CPs, respectively, and both met the required precision according to the corresponding surveying and mapping standards of China. The textural quality of both 3D models and DOMs were distinguishable and could represent the actual color of different ground objects, which indicated that our smartphone-based OP system had the capability of capturing images with high quality. These application examples confirm the quality and accuracy of our system. Although the flight efficiency and the accuracy of our UAV OP system were lower than that of the commercial one, it has many distinctive merits of low-cost, ultralight, and easier maintenance and operation. The smartphone-based multicamera imaging system was then installed in the existing DJI UAV systems. However, its main functions of the automatic capture control and synchronously recording the GPS/INS data were not used and included in our UAV OP system, simply because this kind of UAV system had no extra ports to use. In other words, the automatic capture control of our system was not directly controlled by DJI P4P’s flight control system, and we replaced it by developing an independent module, namely, automatic imaging control. It was expected that the performance of the flight control system would improve continuously, which, in turn, would enhance the feasibility of our smartphone-based multicamera system as payloads for the OP system in the future.

Data Availability

The data used to support the findings of this study are currently under embargo while the research findings are commercialized. Requests for data, 6/12 months after the publication of this article, will be considered by the corresponding author.

Conflicts of Interest

The authors declare that there are no conflicts of interest.

Authors’ Contributions

All authors have contributed equally to the work. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

This research was funded by the National Key R and D Program of China (2017YFE0119600) and the Chinese National Natural Science Foundation (41371438).

References

I. Colomina and P. Molina, “Unmanned aerial systems for photogrammetry and remote sensing: a review,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 92, pp. 79–97, 2014.
View at: Publisher Site | Google Scholar
H. Chao, Y. Cao, and Y. Chen, “Autopilots for small unmanned aerial vehicles: a survey,” International Journal of Control, Automation and Systems, vol. 8, no. 1, pp. 36–44, 2010.
View at: Publisher Site | Google Scholar
M. Nagai, C. Tianen Chen, R. Shibasaki, H. Kumagai, and A. Ahmed, “UAV-borne 3-D mapping system by multisensor integration,” IEEE Transactions on Geoscience and Remote Sensing, vol. 47, no. 3, pp. 701–708, 2009.
View at: Publisher Site | Google Scholar
A. C. Watts, V. G. Ambrosia, and E. A. Hinkley, “Unmanned aircraft systems in remote sensing and scientific research: classification and considerations of use,” Remote Sensing, vol. 4, no. 6, pp. 1671–1692, 2012.
View at: Publisher Site | Google Scholar
J.-Y. Rau, J.-P. Jhan, and Y.-C. Hsu, “Analysis of oblique aerial images for land cover and point cloud classification in an urban environment,” IEEE Transactions on Geoscience and Remote Sensing, vol. 53, no. 3, pp. 1304–1319, 2015.
View at: Publisher Site | Google Scholar
A. Jaakkola, J. Hyyppä, A. Kukko et al., “A low-cost multi-sensoral mobile mapping system and its feasibility for tree measurements,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 65, no. 6, pp. 514–522, 2010.
View at: Publisher Site | Google Scholar
W. Gruszczynski, W. Matwij, and P. Cwiakała, “Comparison of low-altitude UAV photogrammetry with terrestrial laser scanning as data-source methods for terrain covered in low vegetation,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 126, pp. 168–179, 2017.
View at: Publisher Site | Google Scholar
Z. Xu, J. Yang, C. Peng et al., “Development of an UAS for post-earthquake disaster surveying and its application in Ms7.0 lushan earthquake, sichuan, China,” Computers & Geosciences, vol. 68, pp. 22–30, 2014.
View at: Publisher Site | Google Scholar
U. Niethammer, S. Rothmund, M. R. James, J. Travelletti, and M. Joswig, “UAV-based remote sensing of landslides,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 38, no. 5, pp. 496–501, 2010.
View at: Google Scholar
J. Wang, L. Wang, M. Jia, Z. He, and L. Bi, “Construction and optimization method of the open-pit mine DEM based on the oblique photogrammetry generated DSM,” Measurement, vol. 152, Article ID 107322, 2020.
View at: Publisher Site | Google Scholar
P. Sestras, S. Roșca, Ș. Bilașco et al., “Feasibility assessments using unmanned aerial vehicle technology in heritage buildings: rehabilitation-restoration, spatial analysis and tourism potential analysis,” Sensors, vol. 20, no. 7, p. 2054, 2020.
View at: Publisher Site | Google Scholar
H. Liang, W. Li, S. Lai, L. Zhu, W. Jiang, and Q. Zhang, “The integration of terrestrial laser scanning and terrestrial and unmanned aerial vehicle digital photogrammetry for the documentation of Chinese classical gardens - a case study of huanxiu shanzhuang, suzhou, China,” Journal of Cultural Heritage, vol. 33, pp. 222–230, 2018.
View at: Publisher Site | Google Scholar
F. Neitzel and J. Klonowski, “Mobile 3D mapping with a low-cost UAV system,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 38, no. 1/C22, pp. 39–44, 2011.
View at: Publisher Site | Google Scholar
S. Mentasti and F. Pedersini, “Controlling the flight of a drone and its camera for 3D reconstruction of large objects,” Sensors, vol. 19, no. 10, p. 2333, 2019.
View at: Publisher Site | Google Scholar
B. Wu, L. Xie, H. Hu, Q. Zhu, and E. Yau, “Integration of aerial oblique imagery and terrestrial imagery for optimized 3D modeling in urban areas,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 139, pp. 119–132, 2018.
View at: Publisher Site | Google Scholar
Y. Furukawa and J. Ponce, “Accurate, dense, and robust multiview stereopsis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 8, pp. 1362–1376, 2010.
View at: Publisher Site | Google Scholar
G. J. Grenzdörffer, M. Guretzki, and I. Friedlander, “Photogrammetric image acquisition and image analysis of oblique imagery,” Photogrammetric Record, vol. 124, pp. 372–386, 2008.
View at: Google Scholar
J. Xiao, M. Gerke, and G. Vosselman, “Building extraction from oblique airborne imagery based on robust façade detection,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 68, pp. 56–68, 2012.
View at: Publisher Site | Google Scholar
K. Svennevig, P. Guarnieri, and L. Stemmerik, “From oblique photogrammetry to a 3D model - structural modeling of kilen, eastern north Greenland,” Computers & Geosciences, vol. 83, pp. 120–126, 2015.
View at: Publisher Site | Google Scholar
H. Hainan Cui, S. Shuhan Shen, W. Wei Gao, and Z. Zhanyi Hu, “Efficient large-scale structure from motion by fusing auxiliary imaging information,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3561–3573, 2015.
View at: Publisher Site | Google Scholar
S. Shuhan Shen, “Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes,” IEEE Transactions on Image Processing, vol. 22, no. 5, pp. 1901–1914, 2013.
View at: Publisher Site | Google Scholar
S. Harwin and A. Lucieer, “Assessing the accuracy of georeferenced point clouds produced via multi-view stereopsis from unmanned aerial vehicle (UAV) imagery,” Remote Sensing, vol. 4, no. 6, pp. 1573–1599, 2012.
View at: Publisher Site | Google Scholar
N. Micheletti and J. H. ChandlerJ. H. Chandler and S. N. Lane, “Investigating the geomorphological potential of freely available and accessible structure-from-motion photogrammetry using a smartphone,” Earth Surface Processes and Landforms, vol. 40, no. 4, pp. 473–486, 2015.
View at: Publisher Site | Google Scholar
S. Tavani, P. Granado, U. Riccardi, T. Seers, and A. Corradetti, “Terrestrial SfM-MVS photogrammetry from smartphone sensors,” Geomorphology, vol. 367, Article ID 107318, 2020.
View at: Publisher Site | Google Scholar
R. Shults, P. Krelshtein, O. Kravchenk, O. Rogoza, and O. Kyselow, “Low-cost photogrammetry for culture heritage,” in Proceedings of the International Conference on Environmental Engineering, pp. 1–8, Vilnius, Lithuania, April 2017.
View at: Publisher Site | Google Scholar
J. Kim, S. Lee, H. Ahn, D. Seo, S. Park, and C. Choi, “Feasibility of employing a smartphone as the payload in a photogrammetric UAV system,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 79, pp. 1–18, 2013.
View at: Publisher Site | Google Scholar
K. Yang, G. Y. Yang, and S. I. Huang Fu, “Research of control system for plant protection UAV based on pixhawk,” Procedia Computer Science, vol. 166, pp. 371–375, 2020.
View at: Publisher Site | Google Scholar
L. Meier, P. Tanskanen, L. Heng, and G. H. LeeF. Fraundorfer and M. Pollefeys, “A micro aerial vehicle design for autonomous flight using onboard computer vision,” Autonomous Robots, vol. 33, no. 1–2, pp. 21–39, 2012.
View at: Publisher Site | Google Scholar
H. H. Helgesen, F. S. Leira, T. H. Bryne, S. M. Albrektsen, and T. A. Johansen, “Real-time georeferencing of thermal images using small fixed-wing UAVs in maritime environments,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 154, pp. 84–97, 2019.
View at: Publisher Site | Google Scholar
Y. Zhang, C. Peng, D. Mou, M. Li, and W. Quan, “An adaptive filtering approach based on the dynamic variance model for reducing MEMS gyroscope random error,” Sensors, vol. 18, no. 11, pp. 3943–3957, 2018.
View at: Publisher Site | Google Scholar
Z. Chen, Q. Zhu, and Y. C. Soh, “Smartphone inertial sensor-based indoor localization and tracking with ibeacon corrections,” IEEE Transactions on Industrial Informatics, vol. 12, no. 4, pp. 1540–1549, 2016.
View at: Publisher Site | Google Scholar
S. Zhang, D. Hou, C. Wang, F. Pan, and L. Yan, “Integrating and managing BIM in 3D web-based GIS for hydraulic and hydropower engineering projects,” Automation in Construction, vol. 112, Article ID 103114, 2020.
View at: Publisher Site | Google Scholar
Z. Yang, W. Li, Y. Pei, W. Qiao, and Y. Wu, “Classification of the type of eco-geological environment of a coal mine district: a case study of an ecologically fragile region in western China,” Journal of Cleaner Production, vol. 174, pp. 1513–1526, 2018.
View at: Publisher Site | Google Scholar
L. Liu, L. Chen, and J. Tang, “Present situation and future prospects of geologic environment issues in mines in China,” Disaster Advances, vol. 4, pp. 563–566, 2010.
View at: Google Scholar

Copyright

Copyright © 2022 Huachao Yang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

694

Downloads

643

Citations