Abstract
During the last few years, several approaches have been proposed to improve early warning systems for managing geological risk due to landslides, where important infrastructures (such as railways, highways, pipelines, and aqueducts) are exposed elements. In this regard, an Artificial intelligence Camera Prototype (AiCP) for real-time monitoring has been integrated in a multisensor monitoring system devoted to rock fall detection. An abandoned limestone quarry was chosen at Acuto (central Italy) as test-site for verifying the reliability of the integrated monitoring system. A portion of jointed rock mass, with dimensions suitable for optical monitoring, was instrumented by extensometers. One meter of railway track was used as a target for fallen blocks and a weather station was installed nearby. Main goals of the test were (i) evaluating the reliability of the AiCP and (ii) detecting rock blocks that reach the railway track by the AiCP. At this aim, several experiments were carried out by throwing rock blocks over the railway track. During these experiments, the AiCP detected the blocks and automatically transmitted an alarm signal.
1. Introduction
Gravity-induced instabilities from cliffs represent a topic of interest regarding the monitoring of fast to very fast failures, including rock slides or rock falls from natural and artificial slopes [1]. The relevance of such events is mainly related to the short “advice-time” that should be necessary to transmit alarm signals for early warning in case of exposed infrastructures (such as railway, highways, pipelines, and aqueducts), since no displacement is generally detectable before failure occurrence.
To manage the geological risk that involves main infrastructures, different strategies can be adopted: the first one (focused on hazard) consists in monitoring precursors of rock failures by micro- or nanoseismometer recording systems as well as by acoustical emission recording [2–5]; the second one (focused on vulnerability) consists in observing the exposed infrastructure by optical devices (among which cameras, laser scanner, and/or interferometer) suitable for detecting fast changes due to objects/obstacles that impact the infrastructure [6–10].
This paper exemplifies an application of the second approach in a test-site designed to host a multisensor monitoring system for detecting rock falls that involve a railway track-target. The experiment aims at evaluating the suitability of an optical device as early warning system for managing infrastructures [11]. Several experiments were carried out using an Artificial intelligence Camera Prototype (AiCP) installed in the Acuto test-site (central Italy) where a railway track was posed close to an unstable rock wall in an abandoned quarry. Main goal of these experiments was detecting obstacles which could reach the track-target, simulating an impact on a railway that threatens train traffic and passenger safety. The tested AiCP is a completely wireless and self-powered device that continuously observes the railway track-target and the near rock cliff to detect possible blocks or debris fallen from the quarry wall and to transmit alarm signals.
The experiments carried out until now at Acuto consisted in inducing rock falls and detecting if the fallen blocks reached the railway track-target. These experiments are part of a project of the Research Center for Geological Risk (CERI) of “Sapienza” University of Rome, in cooperation with the Italian Railway Network (RFI). The project aims at installing the tested monitoring devices along the national railway net, where fast-landslide events can cause obstacles to impact the railway tracks.
The challenge of this research is to integrate the AiCP into a multisensor network for detecting natural fast-landslide processes from precursors to failure.
2. Description of the Optical Monitoring Device
The experienced monitoring approach is based on the innovative AiCP originally developed by the “Istituto di Scienza e Tecnologia dell’Informazione (ISTI)” of the National Research Council of Italy (CNR) [12, 13].
2.1. Hardware and Materials
The AiCP was developed to carry out real-time monitoring applications based on optical vision techniques of change detection analysis. The AiCP is particularly suitable for investigating landslides hazard scenarios and it has enough computational power to accomplish the computer vision task envisaged for monitoring railways.
The device consists in a customized embedded computing board, equipped with an HD webcam. An on-board computer vision algorithm, developed by CNR-ISTI of the National Research Council of Italy, is installed for real-time analysis of the acquired images. The in-site processing results are transmitted through a suitable wireless communication system. The AiCP uses a low-cost technology, very easy to be installed. All hardware components have a low-energy use device and are manageable through open-source Linux libraries and drivers. This allows customization of the prototype according to different purposes.
A main board, managing vision and networking tasks by an integrated wireless communication module (RF Transceiver), is installed on the AiCP. The prototype also contains a power supply system which controls charge and allows setting the optimal energy-saving policies and includes a battery pack and a photovoltaic panel for harvesting energy, guaranteeing the energy autonomy of the monitoring system. The components of the optical monitoring system are summarized in Figure 1.
In the vision board design, an embedded Linux architecture was selected to have a suitable computational power, as well as make the programming easier. Different kinds of Linux-based libraries were evaluated in terms of computing power, flexibility, expandability, price/performance ratio, and technical support. A good solution was achieved by using a Freescale CPU based on the ARM architecture, with support for MMU-like operating systems GNU/Linux. Linux OS are the most stable and compatible with the proposed AiCP. Moreover, compatible libraries are much more customizable. An average consumption, measured at highest speed (454 MHz), guarantees low power performance of AiCP. An on-board step down voltage regulator, featuring high efficiency to ensure a range of voltages from 6 to 25 V very suitable for battery powered system, was included in AiCP. The main board works through several communication interfaces among which there are a RS232 serial port, a Serial Peripheral Interface (SPI) port, and a USB port. It supports some networking interfaces including Ethernet, Wi-Fi, and 3G. For the here presented applications, a 3G communication module has been selected and an USB modem offering speed up to 7.2 Mbits/s has been integrated [12].
The above described main board and the HD webcam are housed into an IP66 cam shield. The HD webcam used for experiments presented here is an HP HD 2300 with 50° FOV in pixel resolution (4 : 3) and 60° FOV in pixel resolution (16 : 9). Another important component of the AiCP (Figure 2) is the LED spotlight (12 V, 10 W, 800 Lumen, 120° of beam angle) that allows operations of the device also at night-time and in low illumination conditions. For this reason, the algorithm has been optimized to guarantee a high resolution of RGB colour images. The use of the LED spotlight does not introduce energy criticality in the system, ensuring continuous operation, even in low lighting conditions. The AiCP hosts a power supply unit (12 V, 65 Ah acid battery and 25 Wp, 21.5 V harvesting module) that allows the complete autonomy of the device powering. Battery voltage is monitored through one Analog to Digital Converter (ADC) installed on the main board, which allows the optimal setting of the energy-savings policies.
2.2. Software and Image Analysis
The AiCP is based on computer vision approaches [13], which concern detection and real-time alert of dangers strictly related to rock blocks or debris that covers the railway track. The image processing is carried out on-board in three steps: (1) background image subtraction, (2) object detection and classification, and (3) object tracking and final data extraction.
The algorithm for change-detection analysis is applied on images acquired in a Region of Interest (RoI) with respect to the total field of view only, that is, where the presence of an obstacle is expected to be detected. Based on this detection, flow information is derived using an innovative tracking algorithm implemented by the CNR-ISTI of the National Research Council of Italy [13] applied in this experiment to preserve both railways and train passenger safety. The algorithm performs a change detection analysis between two consecutive frames in which the background scenario is automatically updated to filter the environmental noise: when the difference exceeds the presetted pixel threshold, the AiCP is able to remotely transmit an alarm signal. The subtraction of the background scenario can be performed also on a specific convex quadrangular RoI. Within the RoI, lightweight detection methods are used to classify a pixel like “modified” (in this case it is assigned to the foreground-image) or “not modified” (in this case it is considered in the background scenario). The two options are selected after a subtraction procedure. More in particular, the analysis of the railway scene and the estimation of the level of service were usually carried out by collecting data about the events of the railway track scene in terms of occurrence, typology, duration, and intensity [14].
The analytic strategy for updating the background scenario guarantees that the RoI does not include external objects. At this pourpose, algorithms featuring adaptive backgrounds are used: the background scenario is constantly updated by merging the old background scenario with new captured images [12]. There are three main solutions to perform the adaptive background strategy, with different levels of computational complexity. The first solution uses an average image in which the background scenario is on-board modelled by averaging the captured frames within a time window, though an equal threshold for all the pixels. The second solution applies a statistical model of the background scenario including the standard deviation of pixel intensities, that is, considering the dataset of pixel properties as a single Gaussian distribution, where a constant threshold is used based on the probability that an observed pixel could represent a sample drawn from background distribution. The statistical approach also includes the Gaussian Mixture Models (GMM), consisting in merging Gaussian distributions obtained for different pixels instead of using a unique Gaussian distribution obtained for each pixel. GMM allows performing a change detection analysis on background scenario also in case of occurrence of disturbance processes (i.e., trembling shadows or tree foliage that generally can perturbate the monitored scenario) [15]. The codebooks strategy [16] is a third solution for adaptive background modelling that presents computational advantages for real-time background analysis with respect to GMM. Sample background values at each pixel are quantified in codebooks, which represent a compressed form of background model for long image sequences. This allows to capture each complex structural background variation in a long-time window. In the AiCP several ad hoc procedures, implemented by CNR-ISTI of the National Research Council of Italy, can be expected starting from the above described approaches. The background scenario is modelled on the recorded frames in a specific time window by performing an integrated on-board computational background adaptive analysis. As a result, a certain pixel is considered like “modified” if it exceeds a given threshold (that is unique for the entire pixel) with respect to the corresponding pixel in the background scenario [14].
The AiCP is designed to perform computer vision analysis in real-time. To this aim, the approach adopted consists in analyzing the RoIs corresponding to an alert area (i.e., rock walls corresponding to trenches along the railway) and an alarm area (i.e., railway tracks where possible obstacles can impact).
The algorithm, implemented by CNR-ISTI of the National Research Council of Italy, is based on the general motion paradigm, applied only to the area under control, and it is calibrated for real-time monitoring. More in detail, the algorithm analyses both the RoI of alert and the RoI of alarm, so performing a change detection analysis between the actual frame and a dynamically update background scenario [13]. The algorithm checks frames within the RoI of alert at time (named ) and compares them with a dynamic background frame , which is constantly updated as a weighted mixture of and [13]:where is a weight computed in the range [0⋯1]. The pixel threshold for the comparison and motion detection needs to be calibrated: depending on dimensions and typology of the obstacle to be detected. Once the alert is given (i.e., threshold overcome for at least consecutive frames) the control system focuses on the real railway monitoring target (RoI of alarm) where the same algorithm is applied and the same dynamic policy for updating background scenario is adopted. Possible obstacles are expected to reach the railway track within certain number of “waiting frames” (i.e., time needed for the obstacle to pass from the RoI of alert to the RoI of alarm). These waiting frames depend on the specificity of the case study (i.e., the distance between RoI of alert and RoI of alarm or the distance of the camera from the monitored area). If the expected time expires but no obstacles are detected on the railway track, a false alarm is sent. If obstacles have been detected on the railway track, the algorithm continuously monitors motion in the RoI of alarm, by performing a change detection analysis between the present frames and the previous one and carring out a comparative analysis with the assumed pixel threshold. If obstacles move inside the RoI of alarm, the algorithm detects it and it is continuously monitored over frames. If obstacles come out of the RoI of alarm, nothing is changed with respect to the initial monitored scenario. If obstacle stops moving inside the RoI of alarm, after frames, the check on the RoI is performed by detecting a difference of motion between the present frame and the previous one . A comparison between the check results and the considered threshold is performed. If a detected object is endangering the real railway monitoring area, an alert signal is transmitted (Figure 3) [13].
(a)
(b)
(c)
The functional flow of the monitoring system (Figures 2 and 3) is based on a cloud for security purpose and data replication. The AiCP forwards optical data and images to Internet server configured by LAMP (Linux, Apache, MySQL, and PHP) for real-time managing of transmitted data. Three different virtual machines (VM) are hosted on LAMP server and they allow the data collection from AiCP and the data management. The first VM (IoT—Internet of Things) is dedicated to all data intercommunication operations, the second VM (Database) is used for collecting data and provides a cloud database, and the third VM (Web-App) is used for projecting data to the user. Clients can access the front-end, that is, to all applications developed for data management and related alarm signals. Finally, a client application allows data visualization and alarm notification on web and/or mobile phone through specific applications (Figure 4).
3. Experimental Tests in the Acuto Test-Site
Preliminary results were obtained during the 2016 in the Acuto test-site where the AiCP was installed. The multisensor monitoring system installed at Acuto was focused on testing and verifying the reliability of the AiCP for detecting rock block or debris over the railway track-target.
Three experimental goals were pursued at Acuto for testing the AiCP device: (i) detecting obstacle (i.e., rock block and debris) in the RoI of alarm which came from a known direction; (ii) controlling the RoI of alert to identify movement of objects which can impact the railway track-target (i.e., entering in the RoI of alarm); (iii) monitoring the two RoIs over a long time (i.e., several months) to check the dynamic updating of the background scenario.
3.1. Acuto Test-Site
A portion of the Acuto quarry wall was selected because of the presence of several joint sets that make the rock mass prone to rock slide or falls, generating blocks that can reach the railway track-target positioned 4 m far from the rock wall.
The quarry area has been excavated in a limestone hill where Mesozoic limestone outcrops [17]. The quarry wall is SE exposed and it is characterized by heights ranging from 15 m up to 50 m (Figure 5).
Geomechanical scanlines and remote surveys with total station have been performed on the quarry wall to provide a geomechanical characterization of the outcropping rock mass. Five joint sets were distinguished: S0 (93/4; dip direction, dip) corresponding to the limestone strata, S1 (131/82), S2 (91/64), S3 (4/80), and S4 (198/86). As it resulted by Point Load tests, the average uniaxial compressive strength of the outcropping rock is 84 MPa. According to Deere & Miller [18], this result corresponds to a stiffness value of about MPa. The rock mass characteristics have been described using RMRb classification [19], assigning a rank of 68 to the Mesozoic limestone. Slope stability analysis allowed us to identify 18 potentially unstable rock blocks, of which average volume is in the order of , on the quarry wall adjacent to the posed railway track-target.
The installed multisensor monitoring system includes (i) weather station equipped with rain-gage, air thermometer, hygrometer, and anemometer for wind speed and wind direction; (ii) geotechnical system including thermometer for rock mass temperature, strain-gages for monitoring microfractures on rock mass, extensometers on open joints for detecting changes in both rock mass parameters, and stress-strain conditions; (iii) seismic monitoring device for detecting possible microseismic precursor events of rock falls on the cliff slope. The AiCP was installed about 13 m far from the quarry wall and about 6 m far from a 1 m length railway track-target specifically provided by the Italian National Railway Agency for this experiment (Figure 6).
All the recorded data are stored in a local CR1000 Campbell Scientific datalogger, set at 1-minute acquisition time. The device is also equipped with an automatic data transmission system, through a GPRS wireless connection on the local server that allows a complete data storage each 4 h and enables a remote control of both the dataset and the setting parameters. All the sensors are recording in continuous mode since March 2016.
All the components of the AiCP were installed on a 4 m height pillar, reproducing a "self-standing configuration." The pillar head hosts one shield containing the main board and the HD webcam of the AiCP, the 10 W LED spotlight, and one 25 Wp photovoltaic panel for powering the system in daylight hours. At the pillar bottom a one IP66 steel box housing a 65 Ah acid battery for powering the system during the night and one solar charge controller to switch power to solar panel from acid battery or vice versa were installed (Figure 7).
The field of view (FOV) of the camera is almost 20 m 15 m extended and two RoIs were defined: (1) a RoI of alert, including the monitored quarry wall, where rock detachments are expected; (2) a RoI of alarm, including only the railway track-target that the fallen blocks are expected to reach (Figure 8).
3.2. Ai Camera Prototype Experimental Tests
The experiments were carried out in three phases: (A) calibration of the pixel threshold requested by the algorithm for alert/alarm; (B) detection of rock blocks reaching the railway track-target in RoI of alarm and control of the RoI of alert to identify possible debris movement towards the railway track-target; (C) monitoring of the two RoIs over a long time (i.e., several months) to carry out a continuously change detection analysis on dynamically update background scenario.
During phase (A), the pixel threshold values used by the algorithm for calculating the minimum detectable block size on the RoIs have been calibrated. These values are a function of rock block size, distance between the AiCP and the monitored target, and pixel resolution of the captured image. An accidental obstruction of the camera FOV can also cause a variation of the algorithm threshold values. To perform the pixel threshold calibration tests, several rock blocks of different size (Figure 9) were manually placed against the railway track-target, that is, within the RoI of alarm, to check the AiCP start recording.
(a)
(b)
The experimental phase (B) was carried out on July of 2016 in order to simulate rock blocks reaching the railway track-target. At this aim, several rock blocks were launched against the railway track-target, through a parabolic trajectory (Figure 10) across the RoI of alert; this activated the change detection algorithm on both RoIs and caused an alarm signal to be transmitted.
(a)
(b)
(c)
The obtained results prove the high-quality performance of AiCP for detection since all the launched blocks were reported (Figure 11). In only four cases, the blocks were not detected because of the following three reasons: (1) size of the blocks below the algorithmic functionality thresholds, (2) blocks crumbled after the impact with the railway track-target reducing their size below the detectable threshold, and (3) blocks crossing the RoI of alarm and stopping outside of it. In these cases the AiCP did not detect the rock block (i.e., case 1) or detect it but did not correctly launch the alarm signal (i.e., cases 2 and 3).
(a)
(b)
(c)
Another experiment regarded the detectability of fallen blocks over the railway track-target at night-time, that is, with the LED spotlight operating. The results demonstrate that the LED spotlight allows the detection algorithm to be operative. Nevertheless, in night hours a histogram adjustment on the acquired image was adopted and new threshold values for detection have been set applied. Based on the results obtained so far (Figure 12), the dynamic variation of the detection threshold induces negligible effects in terms of false alarms since they can be minimized without decreased sensitivity.
(a)
(b)
(c)
(d)
The experiments performed at the Acuto quarry test-site were also focused on the efficiency and durability of the whole assembled device. The used hardware components resulted suitable to a continuous use in open field. Moreover, the remote data transmission has been allowed by a commercial Subscriber Identity Module (SIM chart). Alarm signals, and the annexed image frames, were transmitted with regularity. The energetic efficiency of the power supply unit was also tested by periodic manual checks of the lithium battery voltage, carried out during the daily hours as well as during different experimental phases (Figure 13). During the daylight hours, the photovoltaic panel always powered the AiCP so that the excess of voltage allows the lithium battery to be charged and to power the AiCP device during the night and/or when daylight conditions did not allow the efficiency of the photovoltaic energy supply unit. At this aim, the photovoltaic panel and the lithium battery are adapt to overbalance each another with respect to the energetic demand of the AiCP device.
4. Conclusions
An Ai Camera Prototype (AiCP) device, developed by CNR-ISTI of the National Research Council of Italy, was tested to detect obstacles reaching railway tracks from rock walls. At this aim, it was installed in an abandoned quarry at Acuto (Italy) where several experiments were performed to check the device reliability and efficiency to detect rock blocks over a railway track-target and to transmit alarm signals.
The experiment results demonstrate the following:(a)the suitability of the AiCP device in the site-test conditions for detecting obstacles over a railway track-target;(b)the flexibility of the operating pixel threshold, which has been calibrated taking into account the rock block size, for performing change detection analysis;(c)the reliability of the detection algorithm for detecting rock blocks down to 20 cm length;(d)the reliability of the network transceiver installed on the main board of the AiCP for transmitting alarm signals, resulting from on-board change detection analysis;(e)the suitability of the LED spotlight that allowed the AiCP device to operate in continuous mode, also during night hours and/or in adverse weather conditions;(f)the efficiency of the power supply unit through periodically voltage check.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The authors wish to thank the CNR-ISTI research team (O. Salvetti, G. R. Leone, M. Magrini, D. Moroni, G. Pieri, and M. Tampucci) for providing the AiCP and the on-board algorithm as well as for their technical support during the experimental activities carried out in the framework of an agreement with CERI Research Centre of the University of Rome “Sapienza”; the Municipality of Acuto for the authorization provided to these experiments; the Italian Rail Network (RFI) for technical support. This study is part of an agreement between CERI Research Centre of the University of Rome “Sapienza” and Italian National Railway Agency (scientific responsible: Professor Salvatore Martino) and also of academic project of the University of Rome “Sapienza” for the year 2016 (UA.S.047-DST): “Cliff Slope Failures in the Coastal Morpho-Dynamics: From Back- to Forward-Analysis of Processes through Monitoring and Multimodelling Approaches” (scientific responsible: Professor Salvatore Martino). The experiments were carried out in the framework of a Ph.D. research (Matteo Fiorucci) and of a research fellowship (Andrea Fantini) funded by the CERI.