Abstract
Currently, due to the rapid development and popularization of smartphones, the usage of ubiquitous smartphones has attracted growing interest in the field of structural health monitoring (SHM). The portable and rapid cable force measurement for cable-supported structures, such as a cable-stayed bridge and a suspension bridge, has an important and practical significance in the evaluation of initial damage and the recovery of transportation networks. The extraction of dynamic characteristics (natural frequencies) of cable is considered as an essential issue in the cable force estimation. Therefore, in this study, a vision-based approach is proposed for identifying the natural frequencies of cable using handheld shooting of smartphone camera. The boundary of cable is selected as a target to be tracked in the region of interest (ROI) of video image sequence captured by smartphone camera, and the dynamic characteristics of cable are identified according to its dynamic displacement responses in frequency domain. The moving average is adopted to eliminate the noise associated with the shaking of smartphone camera during measurement. A laboratory scale cable model test and a pedestrian cable-stayed bridge test are carried out to evaluate the proposed approach. The results demonstrate the feasibility of using smartphone camera for cable force estimation.
1. Introduction
Since cables are primary elements for ensuring the overall structural integrity and safety of cable-stayed bridges, the accurate cable force measurement has very important practical significance in the bridge health monitoring [1]. The cable-stayed bridges are monitored by the continuous cable force measurements during construction, and the bridge health status is evaluated by the change in cable force resulting from the damage and degradation during service stage. In particular, the portable and rapid cable force measurement plays an extremely vital role in the evaluation of initial damage and the formulation of recovery plans for ensuring the security of transportation networks after severe events such as typhoon and earthquake.
Currently, the vibration method is widely used for the estimation of cable force because of its simplicity and speediness [2, 3]. In the vibration method, the cable force is indirectly estimated from the natural frequencies of cable [4, 5], which are commonly extracted from the acceleration responses of cable that is measured by accelerometer sensor installed on cable. The vibration method, however, requires that each of accelerometer sensors should be connected with data acquisition system through electrical cables, and the cable installation is costly and time-consuming. Although wireless sensors have recently emerged to overcome some of these issues, additional issues are then introduced concerning the data transmission reliability, power supply, and network bandwidth [6]. Therefore, it is necessary to develop noncontact method for estimating cable force without attaching sensors to cables.
As one of the typical noncontact methods, vision-based method, which extracts dynamic characteristics of object from video images through the template matching techniques, has received increasing attention in SHM such as system identification [7–12] and damage detection [13, 14]. Although the vision-based method is largely influenced by outdoor field environmental conditions such as illumination variation and background disturbance, it offers significant advantages such as low cost, ease of operation, and multipoint vibration measurement with a single camera. To our knowledge, only a few attempts have been made to apply vision-based methods for cable force estimation. For example, Ji and Chang [15] used a nontarget image-based technique for cable vibration measurement. An optical flow method was adopted to estimate displacement of an arbitrarily chosen ROI on the image plane. The diameter of the cable and the coordinates of the camera were adopted as references for determining the actual displacement of ROI in the 3D object space. The optical flow method considers the motion of object at pixel level according to the change in the image intensities on the image plane [16] and requires an iterative approach [17] for calculating optical flow vector that indicates the magnitude and direction of motion of the ROI in the image plane. Kim et al. [18] developed a vision-based monitoring system for measuring the dynamic characteristics of remote stay cables by means of the template matching method using the normalized cross correlation. The template matching method makes it possible for the object itself to be used as a template in the case of no target.
On the other hand, with the rapid development and popularization of smartphone in recent years, the usage of ubiquitous smartphone has attracted growing interest in the field of SHM owing to its unique feature [19]. Most smartphones have powerful operating system, large memory resource, and a variety of high-performance sensors, and such characteristics are very appropriate for the implementation of SHM. Several researchers studied the application feasibility of smartphone for extracting structural vibration responses from the accelerometer embedded in smartphone [20–24]. Particularly, the smartphone application Orion-CC developed by the authors for smartphone based cloud-SHM [25] is possible to estimate the cable force using the accelerometer embedded in smartphone as well as to upload to the data sharing platform and to share with the public [26]. Although the smartphone inner sensors have limitation in precision of measurement, they are appropriate for the cable force estimation because cable is relatively distinct in vibration rather than the other structural elements. In the previous study, the authors investigated the feasibility and validity of vision-based cable force measuring method using smartphone camera [27]. The dynamic characteristics of cable were extracted from the displacement response of a predefined black circle target attached on the cable. The laboratory cable model test demonstrated the feasibility and reliability for extracting dynamic characteristics of cable at the sampling rate (frames per second) of the current smartphone camera.
As can be seen from the previous studies, the vision-based cable force monitoring is mainly conducted by the vision system (including a digital camera installed with high-quality zoom lens, a computer for image processing, and a data storage server) mounted on the deck or tower of the cable-stayed bridge. Moreover, the dynamic characteristics (natural frequencies) of cable are extracted from the actual real-time displacement responses of cable. Meanwhile, the cable force estimation using smartphone is limited to the smartphone inner accelerometer, and in the case of using smartphone camera, the dynamic characteristics of cable are extracted from the actual displacement responses of a predefined target attached on the cable surface, by means of smartphone mounted on tripod. The utilization of smartphone for measuring cable force would make it possible for personnel to evaluate quickly the initial damage and safety status of cable-stayed bridge, owing to its features such as ease of installation, low cost, and convenience. Although the cable force estimation based on the smartphone inner accelerometer is convenient and easy to install, it requires that the smartphones should be fixed tightly on the cables using the mobile phone jackets because the coupling between the cable and the smartphone can affect the vibration measurement, which sometimes may be either difficult or inappropriate for cable force measurement of inaccessible cable bridges. Therefore, it is necessary to develop a portable and noncontact method for estimating cable force using smartphone camera without any target and tripod.
In this study, as the update of the target vision-based cable force measurement using a smartphone, a novel nontarget approach that is appropriate for handheld shooting of smartphone camera is proposed and its feasibility and practicality are preliminarily validated through a laboratory test and a field test. The dynamic characteristics for cable force estimation are identified from the dynamic displacement responses of one side of cable boundary on the image plane. To enhance the accuracy of cable force measurement, the correction for the shaking of smartphone camera during measurement is performed by means of the moving average filter in the analysis process of cable vibration. The proposed method does not require any installation of tripod and target unlike the previous one. Furthermore, the ROI is selected on the cable boundary through the detection of it, and the cable force is estimated via smartphone application without the calibration for the actual displacement response. The comprehensive comparison tests between using smartphone built-in accelerometer and using smartphone camera, including target and nontarget method, fixed and handheld shooting, and smartphone application and MATLB postprocessing, demonstrate the feasibility and practicality of using smartphone camera for cable force estimation.
2. Cable Force Estimation Algorithm Using Smartphone Camera
2.1. Image Processing Method
2.1.1. Hough Transform for Detecting Cable Boundary
In order to convert 2D vibration of cable on image plane to 1D vibration in accordance with the data processing performance of the current smartphone, it is necessary to define the inclination angle of cable for the rotation transform of image, which can be determined by the position of cable boundary in the image plane. The boundary of cable for identifying the inclination angle of inclined stay cable and for selecting the ROI is detected by Hough transform (HT) on the image of the first scene captured by smartphone camera. HT is recognized as a popular and efficient technique for detecting straight lines in images, even in the presence of noise [28–32]. The Canny edge detection algorithm based on the Gaussian low-pass filter [33, 34] is first adopted to identify all of the edges contained in the grayscale image, resulting in a binary image. By means of HT, for each chosen value of , the edge pixels of image are voted to a parameter space according to where is the distance from the origin to the line along a vector perpendicular to the line and is the angle between the line’s perpendicular and the horizontal axis, and then the parameters of lines are determined by identifying the highest values (i.e., peaks) in the accumulator array (, ).
2.1.2. Measurement of Cable Boundary Location
Since the amplitude of cable vibration is relatively small compared with the distance between smartphone camera and cable, the actual motion of cable in the 3D object space and the corresponding motion on the 2D image plane are linearly correlated, and thus they have the same frequency contents obtained from Fourier transform. In addition, since only the vibration response (i.e., frequencies) of cable is required in the cable force estimation, there is no need to determine the scaling factor (units: mm/pixel) to transform the pixel coordinate vibrations into physical coordinate ones, and the dynamic characteristics of cable can be extracted according to the change in the location of the cable boundary in the chosen ROI of image. The boundary of cable in the ROI is represented as an edge between two image regions as shown in Figure 1.
Assuming an ideal negative edge profile centered in ,where and represent the maximum and minimum intensity values [35]. The image intensity profile is obtained as the convolution of the function and a Gaussian function with zero mean and standard deviation as shown in (3b):This edge can be easily detected using the first-order derivative of the image intensity profile function; namely, the location of the cable boundary in the ROI is identified as the location of the maximum first-order finite-difference in each of the frames. The sums of the pixel intensities in the ROI are used to estimate the dynamic characteristics of cable because the pixel intensity of one point is not sufficient for the accurate location determination of the cable boundary.
2.2. Moving Average Filter
To estimate the dynamic characteristics of cable using the handheld shooting of smartphone camera without a tripod, it is necessary to reduce the noise associated with the shaking of smartphone camera during measurement. In general, the process of determining position and orientation of cameras such as SLAM (Simultaneous Localization and Mapping) or Visual Odometry requires a great deal of computational power and a high calculation cost; it would be inappropriate for the applications of the current smartphones. When the shaking of the smartphone camera is relatively weak compared to the vibration of cable, the moving average can be used to extract the movement components of smartphone from the displacement-time history. Generally, the excellent quality filter in the frequency domain could cause the worst quality in the time domain and vice versa [36]. Although the moving average is insufficient to classify one band of frequencies, it is considered as the most popular filter in Digital Signal Processor (DSP), because it not only is easy to comprehend and use but also possesses exceptionally good property in the time domain. It is commonly used to smooth out short-term variations and emphasize longer-term trends or cycles. The moving average filter controls accuracy of the dynamic characteristics by averaging a number of points from the input data to obtain each point in the output data as expressed in where is the input data, is the output data, and is the number of points in the average. The more accurate displacement-time history for extracting dynamic characteristics of cable is obtained by removing the movement components of smartphone that are smoothed out through the moving average, prior to the Fourier transform.
2.3. Summary of Algorithm
The algorithm for cable force estimation using smartphone camera is shown in Figure 2. The video file captured using smartphone camera is converted into image files. The boundary of cable is detected by means of Hough transform (HT) on the image of the first scene captured by smartphone camera, from which the inclination angle of cable is determined, and the ROI is selected and rotated around its centroid accordingly on each of images converted from video file. The sums of the pixel intensities along the longitudinal direction of cable and its first-order finite-differences are calculated to identify the location of the cable boundary in the ROI. The integer-pixel location of the cable boundary is acquired by locating the peak of the first-order finite-differences, and the more accurate location in less than one pixel is identified in a neighborhood around the initial peak using the first-order Taylor series approximation. The displacement-time history of cable boundary is obtained according to the change of peak in its location and then smoothed out by means of the moving average in order to eliminate the noise generated by shaking of smartphone.
The power spectral density (PSD) is obtained from the displacement response via FFT, from which the natural frequencies in each mode are extracted using the peak picking method. The cable force is finally estimated by the relationship between the frequency difference and cable force as shown inwhere is the linear mass density, is the cable length, and is the frequency difference. Although several practical formulae for cable force estimation have been studied, the flat taut string theory that neglects the effect of sag and stiffness of cable is adopted to evaluate the feasibility of cable force measurement using smartphone camera, due to its simplicity and speediness. In this study, a smartphone application that is developed by Objective-C in accordance with the algorithm explained above is used to estimate the cable force by the smartphone camera.
3. Experimental Verifications
To validate the proposed cable force measurement method based on the image processing, two tests are carried out, as compared with those of using smartphone built-in accelerometer, including a laboratory scale cable model test and a field test. In each test of cable force measurement using smartphone camera, the comparison tests are conducted in three different ways, namely, the measurement using smartphone mounted on a tripod when there is an artificial target, those using smartphone mounted on a tripod when there is no any target, and those using handheld shooting of smartphone camera without any target and tripod. As the smartphone for experiment, four iPhone 6 devices are used. The first smartphone referred to as Smartphone 1, in which a smartphone application Orion-CC is installed, is used to estimate cable force using smartphone built-in accelerometer, and its sampling rate is set to 100 Hz. The smartphone application Orion-CC is possible to not only estimate cable force from the acceleration responses of cable vibration that is extracted using accelerometer embedded in smartphone, but also upload to the data sharing platform and share with the public. Further details regarding the operation guide and cable force measurement of the smartphone application Orion-CC can be seen in [22]. The second smartphone referred to as Smartphone 2, in which a smartphone application D-Viewer is installed, is utilized for the extraction of dynamic characteristics of cable from the dynamic displacement response of a black circle target attached on cable. The maximum number of black pixels is determined in the selected ROI of the binary image, from which the centroid of target and its dynamic displacement response are extracted for cable force measurement. The sampling rate of D-Viewer is 30 Hz; namely, it is the same as the frames per second (fps) of smartphone camera. Since D-Viewer is a vision-based displacement monitoring application using smartphone camera, it requires the actual size of target and the calibration for the actual displacement response. Further details regarding the operation guide and structural displacement monitoring of the smartphone application D-Viewer can be found in [37]. The other two smartphones referred to as Smartphone 3 and Smartphone 4 are used to estimate the cable force using the proposed image processing approach, respectively. One is mounted on the tripod, while the other one is used for the handheld shooting of any participant. Table 1 summarizes the relevant parameters of iPhone 6 used in experiments [38].
To verify the accuracy and precision of the identified natural frequencies, error analysis is carried out using the mean absolute deviation percent (MADP) shown in (6) and the root mean square error (RMSE) shown in (7) where is the number of replicates, are the natural frequencies measured by smartphone built-in accelerometer and smartphone camera mounted on the tripod and those (handheld shooting) processed by MATLAB, respectively, and are the natural frequencies measured by APP D-Viewer, handheld shooting of smartphone camera, and those (handheld shooting) processed by smartphone application, respectively.
3.1. Laboratory Scale Cable Model Test
3.1.1. Overview of Experiment
The cable model is installed in the Bridge Lab of Dalian University of Technology. The model is a 15.53 m single cable, and its linear mass density is 3.95 kg/m. The Smartphone 1 installed with APP Orion-CC is fixed tightly on the cable model via mobile phone jacket, and Smartphones 2 and 3 are mounted on a tripod via double-sided adhesive tape to extract the dynamic characteristics of cable without the shaking of smartphone camera. An organic glass panel, on which there is a preprinted black circle of 2 cm in diameter, is attached to the cable model using superglue. Smartphone 4 is used to measure the cable vibration without any tripod; that is, any participant shoots the cable vibration with it in his hand and extracts the dynamic characteristics of cable. The three smartphones except Smartphone 1 are placed at about 20 cm away from the cable, respectively, so that the field of view appropriate for the measurement is satisfied without smartphone camera zoom. The reference accelerometer is not installed on the cable because the performance of smartphone built-in accelerometer for cable force measurement has been validated through previous studies [22]. Figure 3 shows the installation of smartphones. The smartphone camera captures the cable vibration at a resolution of 1920 × 1080 pixels with a sampling rate of 30 fps. The participant tries to keep a stationary posture as much as possible during measurement. The test is carried out simultaneously under the artificial knocking excitation and the measuring time of the participant is set as 30 seconds taking into account his patience during measurement.
3.1.2. Test Results
The boundary of cable model detected by HT on the image of the first scene captured by smartphone camera, the selected ROI, and the rotation of image are shown in Figure 4. Figures 5–8 show the acceleration and displacement-time history responses of cable model that are measured by four smartphones, and the corresponding power spectral densities (PSD), respectively.
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
(c)
(d)
Figure 8(a) shows the displacement-time history of cable model that the participant measured with Smartphone 4 in his hand, in which the red curve indicates the movement contents of Smartphone 4 smoothed out in time domain by means of the moving average. The displacement-time history of cable model is obtained according to the change in the location of peak in the first-order finite-difference of sums of pixel intensities in ROI, as shown in Figure 8(c). The displacement-time history eliminated the movement contents of smartphone and the corresponding PSD are shown in Figures 8(b) and 8(d), respectively. Figure 9 illustrates three screenshots of smartphone application for cable force measurement using handheld shooting. Table 1 compares the natural frequency values of cable model in each mode of a typical test, which are extracted from PSD by the peak picking method. The error analysis results in the replicates, for the natural frequencies that are measured in different ways using smartphone camera, are summarized in Table 2, which are compared with those using smartphone built-in accelerometer and smartphone application.
(a)
(b)
(c)
As can be seen in the figures, although the vibration amplitudes and PSD peak values are different from each other due to the differences of sampling rate, measuring position, and measuring item, it is observed that the same frequency components can be accurately obtained by the different methods as shown in Table 2. The accuracy in frequency increases as the vibration mode increases, as shown in Table 3. Although the error in frequency changes with different measurement ways, the frequency values extracted by smartphone built-in accelerometer (Smartphone 1) and smartphone camera (Smartphone 4) are highly comparable, with remarkably small error of less than 3.13%.
Furthermore, it is observed that the first natural frequency of cable model is clearly identified by the image processing method using smartphone camera. In general, it is very difficult to extract the first frequency of cable because the measuring position of smartphone is relatively low compared to the length of cable. Although it is impossible to obtain the frequencies in the higher mode owing to the low sampling rate of smartphone camera, the exact frequencies are identified in the lower vibration mode. The excellent agreement is observed between the results of Smartphones 3 and 4, demonstrating the unexceptionally good performance of the moving average for eliminating the shaking of smartphone camera during measurement.
The maximum error in the identified frequencies that are analyzed by means of MATLAB is 2.65%, while those analyzed by the smartphone application have a maximum error of 3.13%. It is observed that the proposed image processing approach is appropriate for the operating system of smartphone.
As a result, the different tests show that the dynamic characteristics of cable for cable force estimation can be accurately identified by the handheld shooting of smartphone, without any target and the information on it, despite the limitation of sampling time, sampling rate, and data processing performance of smartphone.
3.2. Field Test
3.2.1. Overview of Experiment
In order to investigate the feasibility of the vision-based cable force measurement on actual structures, a field test is conducted on a cable-stayed pedestrian bridge located in Dalian City, Liaoning Province, China, as shown in Figure 10. The longest cable is chosen for test at one side of tower. The cable is 21 m in length and 75 mm in diameter. The linear mass density of cable is 18.5 kg/m. Like the laboratory test, the same four smartphones are used to measure cable force in the different ways. To ensure the field of view appropriate for the measurement without smartphone camera zoom, the smartphone cameras shoot the cable vibration at about 30 cm away from the cable, respectively. Again, the preprinted black circle of 2 cm in diameter is used as a target to be tracked by Smartphone 2.
The smartphone camera captures the cable vibration at a resolution of 1920 × 1080 pixels with a sampling rate of 30 fps. The test is carried out simultaneously under the artificial knocking excitation and the measuring time of the participant is set as 30 seconds. The participant also tries to keep a stationary posture as much as possible during measurement. Figure 11 shows the installation of smartphones.
3.2.2. Test Results
Figure 12 shows the cable boundary detected by HT on the image of the first scene captured by smartphone camera, the selected ROI, and the rotation of image. Although the background of image is also very complex, the boundary of cable can be accurately detected by HT for the setting of ROI and the rotation of images. The acceleration and displacement-time history responses of cable that are measured by four smartphones, and the corresponding PSD, are shown in Figures 13–16, respectively. Figure 16(b) is enlarged (between 5 s and 7 s) to show more details on the moving average. Table 4 compares the natural frequencies of cable in each mode that are measured in different ways using four smartphones and the cable forces determined from (5).
(a)
(b)
(a)
(b)
(a)
(b)
(a)
(b)
(c)
(d)
Like the laboratory test, although it is impossible to obtain the frequencies in the higher mode owing to the low sampling rate of smartphone camera, the exact frequencies are identified in the lower vibration mode. Again, the satisfactory agreement is obtained between the cable forces identified using Smartphones 1 and 2, with a small error of less than 1.5%.
The cable forces measured by Smartphones 3 and 4 are highly comparable, demonstrating the unexceptionally good performance of the moving average for eliminating the shaking of smartphone camera during measurement. The error in cable force measured by handheld shooting of smartphone camera is very small (below ±5%). Although the natural frequencies of cable are not accurately identified by the smartphone application in the case of handheld shooting because of the low amplitude of actual structure, the cable forces analyzed by MATLAB agree very well with those of using the smartphone built-in accelerometer.
4. Conclusions
In the study, a vision-based approach is proposed for cable force estimation using handheld shooting of smartphone camera. The boundary of cable is selected as a target, and the displacement-time history of cable is obtained according to the change in its location in ROI of video image sequence captured by smartphone camera. The location of the cable boundary is identified as the location of the maximum first-order finite-difference of the pixel intensity sums in ROI. The noise associated with the shaking of smartphone camera during measurement is eliminated by means of the moving average in time domain. The comprehensive comparison tests are carried out in different measurement ways, involving target and nontarget method, fixed and handheld shooting, and smartphone application and MATLB postprocessing. The following conclusions can be drawn from this study:(1)In spite of the complexity of image background, the boundary of cable is accurately detected on the image of the first scene captured by smartphone camera, and the accurate dynamic characteristics of cable are rapidly identified by only one smartphone camera.(2)The unexceptionally good performance of the moving average for eliminating the shaking of smartphone camera is shown through the comparison with the measurement by the smartphone camera mounted on the tripod.(3)The comparison with the displacement monitoring application demonstrates that it is possible to estimate cable force using smartphone camera without any target and its actual size. Although it is impossible to obtain the frequencies in the higher mode owing to the low sampling rate of smartphone camera, the exact frequencies are identified in the lower vibration mode. The frame rate of smartphone camera is expected to continue to increase in the future.(4)The measurement error of smartphone application in the field bridge cable test shows that the higher-performance smartphone should be used in the cable force measurement.(5)The laboratory and field tests demonstrate the advantages over the traditional measurement methods such as convenience, ease of operation, and speediness.
It is noted that some of the issues are yet to be addressed such as the measurement by environmental and traffic-induced vibration, the remote measurement using smartphone camera zoom lens, the evaluation on shaking of smartphone camera using smartphone built-in sensors, and the application for long-span cable-stayed bridges. Nevertheless, this study demonstrates the feasibility of cable force measurement using handheld shooting of smartphone camera. Based on the preliminary efforts, the authors are exploring the more effective method for cable force estimation using smartphone camera, in accordance with the improvement of smartphone.
Conflicts of Interest
The authors declare no conflicts of interest.
Acknowledgments
The authors acknowledge the National Natural Science Foundation of China for providing the financial support (51278085, 51221961).