Abstract
Potholes can generate damage such as flat tire and wheel damage, impact and damage of lower vehicle, vehicle collision, and major accidents. Thus, accurately and quickly detecting potholes is one of the important tasks for determining proper strategies in ITS (Intelligent Transportation System) service and road management system. Several efforts have been made for developing a technology which can automatically detect and recognize potholes. In this study, a pothole detection method based on two-dimensional (2D) images is proposed for improving the existing method and designing a pothole detection system to be applied to ITS service and road management system. For experiments, 2D road images that were collected by a survey vehicle in Korea were used and the performance of the proposed method was compared with that of the existing method for several conditions such as road, recording, and brightness. The results are promising, and the information extracted using the proposed method can be used, not only in determining the preliminary maintenance for a road management system and in taking immediate action for their repair and maintenance, but also in providing alert information of potholes to drivers as one of ITS services.
1. Introduction
A pothole is defined as a bowl-shaped depression in the pavement surface, and its minimum plan dimension is 150 mm [1]. With the climate change such as heavy rains and snow in Korea, damaged pavements like potholes are increasing, and thus complaints and lawsuits of accidents related to potholes are growing. There are internal causes to potholes such as the degradation and responsiveness or durability of the pavement material itself to climate change, such as heavy rainfall and snowfall, and external causes such as the lack of quality management and construction management.
Also, Table 1 shows the number of compensations and compensation amounts about accidents related to road facilities for 6 years, 2008 to 2013 in Seoul [2].
As shown in Table 1, the number of compensations and compensation amounts related to potholes occupy more than 50% of total the number of compensations and compensation amounts in Seoul city. Seoul city has been pouring attention to prepare a countermeasure of potholes that threaten road safety in this way.
As one type of pavement distresses, potholes are important clues that indicate the structural defects of the asphalt road, and accurately detecting these potholes is an important task in determining the proper strategies of asphalt-surfaced pavement maintenance and rehabilitation. However, manually detecting and evaluating methods are expensive and time consuming. Thus, several efforts have been made for developing a technology that can automatically detect and recognize potholes, which may contribute to the improvement in survey efficiency and pavement quality through prior investigation and immediate action.
Existing methods for pothole detection can be divided into vibration-based methods, three-dimensional (3D) reconstruction-based methods, and vision-based methods [3–26]. Although these vision-based methods are cost-effective compared with 3D laser scanner methods, it may be difficult to accurately detect a pothole using these methods because of the distorted signals generated by noise in collecting image and video data. Thus, a pothole detection method using various features in 2D images is proposed for improving the existing pothole detection method [20] and accurately detecting a pothole. Also, the performance of the proposed method is compared with that of the existing method for several conditions such as road, recording, and brightness. Further, a pothole detection system is designed to be applied to ITS service and road management system. The designed and developed pothole detection system is expected to be used, not only in determining the preliminary maintenance of road management system and in taking immediate action for their repair and maintenance, but also in providing alert information of potholes to drivers as one of ITS services.
2. Literature Review
Several efforts have been made for developing a method which can automatically detect and recognize potholes. Detailed surveys on methods for pothole detection can be found in Koch and Brilakis [20] and Kim and Ryu [27]. Existing methods for pothole detection can be divided into vibration-based methods by B. X. Yu and X. Yu [3], De Zoysa et al. [4], Eriksson et al. [5], and Mednis et al. [6]; three-dimensional (3D) reconstruction-based methods by Wang [7], Kelvin [8], Chang et al. [9], Vijay [10], Hou et al. [11], Li et al. [12], Salari et al. [13], Staniek [14], Zhang et al. [15], Joubert et al. [16], and Moazzam et al. [17]; and vision-based methods by Wang and Gong [18], Lin and Liu [19], Koch and Brilakis [20], Jog et al. [21], Huidrom et al. [22], Koch et al. [23], Buza et al. [24], Lokeshwor et al. [25], and Kim and Ryu [26].
Vibration-based method uses accelerometers in order to detect potholes. Considering the advantages of a vibration-based system, these methods require small storage and can be used in real-time processing. However, vibration-based methods could provide the wrong results, for example, that the hinges and joints on the road can be detected as potholes and that potholes in the center of a lane cannot be detected using accelerometers due to not being hit by any of the vehicle’s wheels (Eriksson et al.) [5].
3D laser scanner methods can detect potholes in real time. However, the cost of laser scanning equipment is still significant at the vehicle level, and currently these works are focused on the accuracy of 3D measurement. Stereo vision methods need a high computational effort to reconstruct pavement surfaces through matching feature points between two views so that it is difficult to use them in a real-time environment [7, 8, 10, 11, 13–15]. Recently, Moazzam et al. [17] used a low-cost Kinect sensor to collect the pavement depth images and calculate the approximate volume of a pothole. Although it is cost-effective as compared with industrial cameras and lasers, the use of infrared technology based on a Kinect sensor for measurement is still a novel idea, and further research is necessary for improvement in error rates.
A 2D image-based approach has been focused only on pothole detection and is limited to a single frame, so it cannot determine the magnitude of potholes for assessment. To overcome the limitation of the above method, video-based approaches were proposed to detect a pothole and calculate the total number of potholes over a sequence of frames.
Although these vision-based methods are cost-effective compared with 3D laser scanner methods, it may be difficult to accurately detect a pothole using these methods because of the distorted signals generated by noise in collecting image and video data. Thus, a pothole detection method using various features in 2D images is proposed for improving the existing pothole detection method [20] and accurately detecting a pothole. Also, the performance of the proposed method is compared with that of the existing method for several conditions such as road, recording, and brightness. In our study, for comparison, the method by Koch and Brilakis [20] was selected because their method had a good result as compared to other existing methods.
3. The Pothole Detection System
A pothole detection system was designed to collect road images through a newly developed optical device mounted on a vehicle and detects a pothole from the collected data using the proposed algorithm. Figure 1 shows a pothole detection system that was developed in this study and its application. This system includes an optical device and a pothole detection algorithm.
The optical device on a vehicle collects potholes data, and the collected data is sent to a pothole detection algorithm. Also, the pothole information such as the location and severity of a pothole obtained from a pothole detection algorithm is sent to a road management center. The optical device was designed to easily be mounted in a vehicle, and it has several functions such as collecting and storing data of potholes, communicating by Wi-Fi, and gathering location information by GPS. Table 2 shows the detailed specification of the optical device.
The pothole information obtained from a pothole detection system is sent to a center and can be applied to a pothole alert service and the road management system. As shown in Figure 2, pothole information is sent from a center to RSEs (Roadside Equipment) and navigation companies, and then the information is sent to OBUs (Onboard Unit) or navigations through DSRC (Dedicated Short-Range Communication) and WAVE communication. Finally, pothole alert information such as location and severity is displayed on OBU or navigation. Before passing the pothole, a driver can recognize the presence of the pothole in advance and avoid risks. Pothole alert service is still a novel idea, and further research is necessary for improvement in image processing time and communication.
Also, the obtained pothole information is provided to the Road Management System, and the repair time and maintenance quantities are determined according to the severity and location of the pothole.
4. The Proposed Pothole Detection Method
The proposed method can be divided into three steps: segmentation, candidate region extraction, and decision (Figure 3). First, a histogram and the closing operation of a morphology filter are used for extracting dark regions for pothole detection. Next, candidate regions of a pothole are extracted using various features, such as size and compactness. Finally, a decision is made whether candidate regions are potholes or not by comparing pothole and background features.
The segmentation step is to separate a pothole region from the background region by transforming an original color image into a binary image using the histogram of an input image. HST (Histogram Shape-Based Thresholding), maximum entropy, and Otsu [28] can be used for this transformation into a binary image. In this study, an input image is transformed into a binary image using HST [20].
The candidate step involves extracting a pothole candidate region from a binary image obtained in the segmentation step. First, the median filter is used to remove noise such as cracks and spots. , , and filters were tested and the filter showed the best performance among the three filters.
Next, the damaged outlines of object regions are restored, and small pieces are removed using the closing operation (dilation and erosion) of a morphology filter. A square () type of the structure element was used for the closing operation.
After the closing operation, candidate regions are extracted using features such as size, compactness, ellipticity, and linearity, as shown inwhere : the value of region for the candidate in the image, : the size of region in the image after the closing operation, : the compactness of region in the image after the closing operation, : the threshold for size, and : the threshold for compactness.
The size of a region is defined as total number of pixels in the region which depends on a size of a pothole, an object distance, and a focal length. Also, compactness is defined as where : the perimeter and : the area of region .
Also, the refinement of candidate regions is needed to detect the correct pothole regions after obtaining the candidate regions. The initial candidates obtained using features may not represent the full-sized pothole area. Thus, the refinement of candidate regions using features such as compactness, center point, and convex hull is necessary before it can be decided whether various and incomplete candidate regions such as shades, spots, and patches are potholes or not. Incomplete candidate regions are refined using the convex hull operation according to the decision ofwhere : the value of refined region for the candidate in the image, : the value of region for the candidate in the image, : the center position of region , : the compactness of region in the image, and : the threshold for compactness.
Next, MHST (modified HST) separates not only the pothole region but also a bright region, such as a lane marking, from the background region.
The decision step involves deciding whether the refined candidate regions are potholes or not after the comparison of candidate regions with the background region using features such as standard deviation and histogram.
In particular, as a histogram feature, ordered histogram intersection (OHI) [29] is used in this study. By using OHI, it is possible to distinguish stains, patches, light, shades, and so forth that cannot be separated from potholes using the existing method [20] and to avoid the wrong detection of potholes. OHI is a method of measuring the degree of similarity between regions in an image. Although some problems occur with noise or when there is a change in brightness, OHI can measure the degree of similarity by identifying these differences. OHI can be expressed as shown inwhere : OHI for candidate region and background region , : the ordered histogram of index for candidate region , : the ordered histogram of index for background region , : the index of histogram ( to 255 for 8 bits), and : the maximum number of the index ( for 8 bits).
In this study, if the standard deviation of the refined candidate region is smaller than the threshold for standard deviation or if OHI of the pixels between the refined candidate region and the background region is close to 1 and the OHI of values using the Sobel operation [30] is close to 1, it is decided that the refined candidate region is not a pothole because it is similar to the background region. Equation (5) shows this discriminant:where : the standard deviation of the refined candidate region , : OHI for the refined candidate region and background region , : OHI for the refined candidate region and background region using the Sobel operation, Outregionstd: the standard deviation of the outside of the refined candidate region, Innerregionstd: the standard deviation of the inside of the refined candidate region, Outregionave: the average of the outside of the refined candidate region, Innerregionave: the average of the inside of the refined candidate region, : the threshold for standard deviation, : the threshold for standard deviation of values by the Sobel operation, : the threshold for average, : the threshold for OHI, and : the threshold for OHI of values by the Sobel operation.
Figure 4 shows the result image at each step by the proposed method.
5. Experiment Results
In this study, 2D road images that had been collected by a survey vehicle in Korea from May to June 2014 were used. Two-dimensional images with a pothole and without a pothole extracted from the collected pothole database (a total of 150 video clips) were used to compare the performance of the proposed method with that of the existing method [20] by several conditions such as road, recording, and brightness conditions.
To collect video data of potholes, the newly developed optical device (resolution 1280 × 720, 60 f/s) were mounted at the height of a rear-view mirror of a survey vehicle, and they recorded the road surfaces ahead during movement.
The proposed pothole detection method was implemented in Microsoft Visual C++ 6.0. The image processing was performed on a laptop (Intel Core i5-4210U, 2.4 GHz, 8 GB RAM). Table 3 shows the values of thresholds used in this study. All threshold values except for (threshold for HST and MHST) were empirically set as the most suitable value to distinguish various types of potholes from similar objects.
A total of 90 images were randomly chosen from 100 video clips for experiments. For experiments by road condition, 20 asphalt images and 20 concrete images were selected randomly, and Figure 5 shows the examples and results of the selected images for experiment by road condition.
(a) Asphalt images
(b) Concrete images
In Figure 5, it is shown that the proposed method accurately detects most of the potholes in both asphalt and concrete images. Fourth image from the left among asphalt images has stains, and the proposed method does not detect them as potholes, but the existing method [20] detects them as potholes.
For experiments by recording condition, 10 original images and 10 images by close-up were selected, and Figure 6 shows the examples and results of the selected images for experiment by recording condition.
(a) Original images
(b) Close-up images
In Figure 6, it is shown that the proposed method accurately detects most of the potholes in close-up images. A few results show that only a portion of the pothole was detected because only that part of the pothole was extracted as a candidate region.
Also, for experiments by brightness condition, 10 bright images (average gray level > 120) and 10 dark images (average gray level < 110) were selected, and Figure 7 shows the examples and results of the selected images for experiment by brightness condition.
(a) Bright images
(b) Dark images
The proposed method has a better performance for bright images, rather than dark images. Not only the proposed method but also all existing methods detect dark regions as suspected potholes. Thus, it is obvious that the performance of detecting potholes under dark circumstances is worse than that of detecting potholes under normal brightness.
In addition, 30 more images for experiments were tested, and the result of pothole detection of experiments using the proposed method and existing method for a total of 90 images are summarized in Table 4. In order to compare the performance of the proposed method with that of the existing method [20], image segmentation and candidate extraction were processed under the same conditions, and the decision criteria for a pothole were applied differently according to the proposed criteria in each method. In this table, in order to represent accurate detection performance, the number of true positives (TP, correctly detected as a pothole), false positives (FP, wrongly detected as a pothole), true negatives (TN, correctly detected as a nonpothole), and false negatives (FN, wrongly detected as a nonpothole) [19] was counted manually. Also, accuracy, precision, and recall using the proposed method and the existing method were calculated as measurements for validation:(1)accuracy: the average correctness of a classification process − (TP + TN)/(TP + FP + TN + FN),(2)precision: the ratio of correctly detected potholes to the total number of detected potholes − TP/(TP + FP),(3)recall: the ratio of correctly detected potholes to actual potholes − TP/(TP + FN).In our study, for comparison, the method by Koch and Brilakis [20] was selected because their method had a good result as compared to other existing methods. Table 4 shows that the proposed method reaches an overall accuracy of 73.5%, with 80.0% precision and 73.3% recall. All three measures validate that most potholes in images can be correctly detected. Also, the results of the proposed method show a much better performance than that of the existing method, which has an overall accuracy of 45.1%, with 55.0% precision and 36.7% recall. By the existing method, it is difficult to separate stains or patches similar to a pothole from an actual pothole using only the feature of standard deviation. However, the proposed method can accurately detect a pothole using several features such as the standard deviation of a candidate region, OHI, differences in the standard deviations, and averages between the outside and inside of a candidate region. It is shown that a joint part between an asphalt road and a concrete road was incorrectly detected. However, this wrong detection can be removed later by adding a feature corresponding to the concrete in the decision step.
Also, the processing times for the proposed method were calculated through 10 of images that were chosen randomly. Table 5 shows the calculated processing times for the proposed method. It is impossible to compare the processing times of the proposed method with those of Koch and Brilakis [20] exactly since it is impossible to implement Koch and Brilakis’ method in their same experiment circumstance and it can result in needing more times for the Koch and Brilakis’ method due to the wrong setting for experiments. However, the processing times of the Koch and Brilakis’ method can be referred to Koch et al. [23].
Table 5 shows that more processing times are needed for Images 3, 7, and 8, since those images have more numbers of candidate regions or bigger regions than other images. It is obvious that the proposed method needs more processing time than Koch and Brilakis [20] because the proposed method uses various features for detecting potholes. Further work for improving image processing time is necessary for the pothole detection system to be applied to real-time pothole detection and real pothole alert service.
The results are promising, and the information extracted using the proposed method can be used, not only in determining the preliminary maintenance for a road management system and in taking immediate action for their repair and maintenance, but also in providing alert information of potholes to drivers as one of ITS services.
6. Conclusions
In this study, a pothole detection method based on 2D road images was proposed for improving the existing method and designing a pothole detection system to be applied to ITS service and road management system. For experiments, 2D road images that were collected by a survey vehicle in Korea were used and the performance of the proposed method was compared with that of the existing method for several conditions such as road, recording, and brightness. Regarding the experiment results, the proposed method reaches an overall accuracy of 73.5%, with 80.0% precision and 73.3% recall, which is a much better performance than that of the existing method having an overall accuracy of 45.1%, with 55.0% precision and 36.7% recall.
However, there are some limitations in the proposed method. Potholes may be falsely detected according to the type of shadow and various shapes of potholes. Thus, in order to more accurately detect potholes, it is necessary to use images from not only a single sensor but also additional sensors and to add to the proposed method more features for these sensors. Also, the stability of the pothole detection method based on two-dimensional images needs to be added, because the vehicle’s vibration during driving will have big affection on the detecting equipment. The proposed method will have a more improved performance through more experiments under a variety of circumstances. In addition, the proposed method needs more processing time than Koch and Brilakis [20] because the proposed method uses various features for detecting potholes. Therefore, further work for improving image processing time and performance of the proposed method is necessary for the pothole detection system to be applied to real-time pothole detection and real pothole alert service.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This research was supported by a grant from a Strategic Research Project (Development of Pothole-Free Smart Quality Terminal [2014-0219]) funded by the Korea Institute of Civil Engineering and Building Technology.