Abstract
Lane detection is a challenging problem. It has attracted the attention of the computer vision community for several decades. Essentially, lane detection is a multifeature detection problem that has become a real challenge for computer vision and machine learning techniques. Although many machine learning methods are used for lane detection, they are mainly used for classification rather than feature design. But modern machine learning methods can be used to identify the features that are rich in recognition and have achieved success in feature detection tests. However, these methods have not been fully implemented in the efficiency and accuracy of lane detection. In this paper, we propose a new method to solve it. We introduce a new method of preprocessing and ROI selection. The main goal is to use the HSV colour transformation to extract the white features and add preliminary edge feature detection in the preprocessing stage and then select ROI on the basis of the proposed preprocessing. This new preprocessing method is used to detect the lane. By using the standard KITTI road database to evaluate the proposed method, the results obtained are superior to the existing preprocessing and ROI selection techniques.
1. Introduction
With the rapid development of society, automobiles have become one of the transportation tools for people to travel. In the narrow road, there are more and more vehicles of all kinds [1]. As more and more vehicles are driving on the road, the number of victims of car accidents is increasing every year [2]. How to drive safely under the condition of numerous vehicles and narrow roads has become the focus of attention. Advanced driver assistance systems which include lane departure warning (LDW) [3], Lane Keeping Assist, and Adaptive Cruise Control (ACC) [4] can help people analyse the current driving environment and provide appropriate feedback for safe driving or alert the driver in dangerous circumstances. This kind of auxiliary driving system is expected to become more and more perfect [5]. However, the bottleneck of the development of this system is that the road traffic environment is difficult to predict [6]. After investigation, in the complex traffic environment where vehicles are numerous and speed is too fast, the probability of accidents is much greater than usual. In such a complex traffic situation, road colour extraction and texture detection as well as road boundary and lane marking are the main perceptual clues of human driving [7].
Lane detection is a hot topic in the field of machine learning and computer vision and has been applied in intelligent vehicle systems [8]. The lane detection system comes from lane markers in a complex environment and is used to estimate the vehicle’s position and trajectory relative to the lane reliably [9]. At the same time, lane detection plays an important role in the lane departure warning system. The lane detection task is mainly divided into two steps: edge detection and line detection.
Qing et al. [10] proposed the extended edge linking algorithm with directional edge gap closing. The new edge could be obtained with the proposed method. Mu and Ma proposed Sobel edge operator which can be applied to adaptive area of interest (ROI) [11]. However, there are still some false edges after edge detection. These errors will affect the subsequent lane detection. Wang et al. proposed a Canny edge detection algorithm for feature extraction [12]. The algorithm provides an accurate fit to lane lines and could be adaptive to complicated road environment. In 2014, Srivastava et al. proposed that the improvements to the Canny edge detection can effectively deal with various noises in the road environment [13]. Sobel and Canny edge operator are the most commonly used and effective methods for edge detection.
Line detection is as important as edge detection in lane detection. With regard to line detection, we usually have two methods which include feather-based method and model-based methods. Niu et al. used a modified Hough transform to extract segments of the lane profile and used DBSCAN (density based spatial application noise clustering) clustering algorithm for clustering [14]. In 2016, Mammeri et al. used progressive probabilistic Hough transform combined with maximum stable extreme area (MSER) technology to identify and detect lane lines and utilized Kalman filter to achieve continuous tracking [15]. However, the algorithm does not work well at night.
In this paper, we propose a lane detection method that is suitable for all kinds of complex traffic situations, especially as driving speed in roads is too fast. First, we preprocessed each frame image and then selected the area of interest (ROI) of the processed images. Finally, we only needed edge detection vehicle and line detection for the ROI area. In this study, we introduced a new preprocessing method and ROI selection method. First, in the preprocessing stage, we converted the RGB colour model to the HSV colour space model and extracted white features on the HSV model. At the same time, the preliminary edge feature detection is added in the preprocessing stage, and then the part below the image is selected as the ROI area based on the proposed preprocessing. Compared with the existing methods, the existing preprocessing methods only perform operations such as graying, blurring, X-gradient, Y-gradient, global gradient, thresh, and morphological closure. And the ways to select the ROI area are also very different. Some of them are based on the edge feature of the lane to select the ROI area, and some are based on the colour feature of the lane to select the ROI area. These existing methods do not provide accurate and fast lane information, which increases the difficulty of lane detection. In this paper, experiments show that the proposed method is significantly better than the existing preprocessing method and ROI selection method in lane detection.
2. Overview of the Proposed System
This paper presents an advanced lane detection technology to improve the efficiency and accuracy of real-time lane detection [16]. The lane detection module is usually divided into two steps: (1) image preprocessing and (2) the establishment and matching of line lane detection model.
Figure 1 shows the overall diagram of our proposed system where lane detection blocks are the main contributions of this paper. The first step is to read the frames in the video stream. The second step is to enter the image preprocessing module. What is different from others is that in the preprocessing stage we not only process the image itself but also do colour feature extraction and edge feature extraction [17]. In order to reduce the influence of noise in the process of motion and tracking, after extracting the colour features of the image, we need to use Gaussian filter to smooth the image. Then, the image is obtained by binary threshold processing and morphological closure. These are the preprocessing methods mentioned in this paper.
Next, we select the adaptive area of interest (ROI) in the preprocessed image. The last step is lane detection. Firstly, Canny operator is used to detect the edge of lane line; then Hough transform is used to detect line lane. Finally, we use Extended Kalman Filter (EKF) to detect and track lane line in real time.
3. Proposed Methods
In this paper, based on the previous preprocessing, we firstly extract the colour features based on the white colour and then extract the edge features based on the straight lane. Because the high-speed section is the traffic accident-prone section, the high-speed road section mostly is the straight line lane [18]. Therefore, in order to obtain a very high recognition rate, we successively carry on colour detection and edge detection to the lane. This paper combines colour features extraction and edge features extraction, and the experiment proves that the recognition rate and accuracy of lane detection are greatly improved.
Our main contribution in this paper is to do a lot of work in the preprocessing stage. We proposed to perform colour transform of HSV in the preprocessing stage, then extract white, and then perform conventional preprocessing operations in sequence. Moreover, we selected an improved method proposed in the area of interest (ROI). In this paper, based on the proposed preprocessing method (after HSV colour transform, white feature extraction, and basic preprocessing), one-half part of the processed image is selected as the area of interest (ROI). In addition, we performed twice edge detection. The first is in the preprocessing stage, and the second is in the lane detection stage after the ROI is selected. The purpose of performing twice edge detection is to enhance the lane recognition rate.
In this paper, Hough transform is used for the straight line detection. Figure 2 shows the basic principles of the Hough transform. In Figure 2(a), each point on the straight line crossing the point and the point corresponds to a straight line and a straight line on the parameter space map in Figure 2(a) after Hough transformation; two lines intersect at the point , where and are the parameters of the line determined by the point and point in Figure 2(a) [19]. On the contrary, the straight line and the straight line where the parameter space in Figure 2(b) intersects at the same point and the collinear points in Figure 2(a) are correspondence [20]. According to this characteristic, given some specific points in Figure 2(a), the line equations connecting these points in Figure 2(b) can be calculated by Hough transform.
The Hough transform is implemented in polar form as [21]
where are coordinates of nonzero pixels in binary image. is the distance between the x-axis and fitted line. is the angle between x-axis and normal line. The value range of is ±90°.
As shown in Figure 3, the Hough transform transforms the points of the image in Figure 3(a) into the polar coordinate parameter space of Figure 3(b). We can see that the collinear point and point in Figure 3(a) intersect at the same point in Figure 3(b). Here, and are the polar parameters of the desired straight line [21].
Different from Figure 2(b), when Figure 3(b) is expressed in polar coordinates, the collinear point and point mapped to the parameter space in the original image intersect at the point [21].
The Kalman filtering algorithm is used to track lane lines in real time. In this paper, we use Extended Kalman Filter (EKF) to track the lane in real time [22]. After the parameters and of the lane based on the straight line model are obtained from the Hough transforms of Figures 2 and 3, the lane line can be tracked using the EKF. The EKF tracking algorithm is described in Table 1, the initial value of the parameter and the initial value of the covariance are set as the unit matrix, and the predicted value of the current state is the tracking result of the previous state [21]. The real value of the current state is the sequence frame of the current reading; thus the tracking value of the current state (the optimal estimation result) can be obtained. This value is also used as the prediction value of the next state to realize the cyclic estimation of the lane parameters, that is, the tracking [23]. Table 1 shows Extended Kalman Filter algorithm module.
As shown in Figure 4, before inputting the image frame, we made preparations such as calculating the average value of vehicle parameter dimensions and setting EKF parameters. The input image frame is detected by the Canny edge operator and the resulting edge image is obtained. Then, we add the parameters and of the lane line based on the straight line model that have been obtained by the Hough transform and determine whether the lane parameters and dimensions detected by the Hough transform are the same for all input frame images. If they are equal, use EKF for lane tracking, or enter the dimension addition and subtraction module to adjust the parameter dimension.
4. Experiments
4.1. Preprocessing
Preprocessing is an important part of image processing and an important part of lane detection. Preprocessing can help reduce the complexity of the algorithm, thereby reducing subsequent program processing time. The video input is a RGB-based colour image sequence obtained from the camera. In order to improve the accuracy of lane detection, many researchers employ different image preprocessing techniques.
Smoothing and filtering graphics is a common image preprocessing technique. The main purpose of filtering is to eliminate image noise and enhance the effect of the image. Low-pass or high-pass filtering operation can be performed for 2D images, low-pass filtering (LPF) is advantageous for denoising, and image blurring and high-pass filtering (HPF) are used to find image boundaries [24–26]. In order to perform the smoothing operation, an average, median [8], or Gaussian [23] filter could be used. In [24], in order to preserve detail and remove unwanted noise, Xu and Li firstly use a median filter to filter the image and then use an image histogram in order to enhance the grayscale image [23].
4.2. Colour Transform
Colour model transform is an important part of machine vision, and it is also an indispensable part of lane detection in this paper. The actual road traffic environment and light intensity all produce noise that interferes with the identification of colour. We cannot detect the separation of white lines, yellow lines, and vehicles from the background. The RGB colour space used in the video stream is extremely sensitive to light intensity, and the effect of processing light at different times is not ideal. In this paper, the RGB sequence frames in the video stream are colour-converted into HSV colour space images. Figures 5(a) and 5(b) are images of RGB colour space and HSV colour space, respectively. HSV represents hue, saturation, and value [6]. As can be seen in Figure 6, the values of white and yellow colours are very bright in the V-component compared to other colours and are easily extracted, providing a good basis for the next colour extraction. Experiments show that the colour processing performed in the HSV space is more robust to detecting specific targets.
(a) RGB
(b) HSV
4.3. Basic Preprocessing
As shown in Figure 7, a large number of frames in the video will be preprocessed. The images are individually gray scaled, blurred, X-gradient calculated, Y-gradient calculated, global gradient calculated, thresh of frame, and morphological closure [25]. In order to cater for different lighting conditions, an adaptive threshold is implemented during the preprocessing phase. Then, we remove the spots in the image obtained from the binary conversion and perform the morphological closing operation. As can be seen from Figure 7, the basic preprocessed frames cannot be very good at removing noise. It can be seen from the results after the morphological closure that although preliminary lane information can be obtained, there is still a large amount of noise.
4.4. Adding Colour Extraction in Preprocessing
In order to improve the accuracy of lane detection, we add a feature extraction module in the preprocessing stage. The purpose of feature extraction is to keep any features that may be lane and remove features that may be nonlane. This paper mainly carries on the feature extraction to the colour. After the graying of the image and colour model conversion, we add the white feature extraction and then carry out the conventional preprocessing operation in turn. The process of the colour extraction proposed in this paper is shown in Figure 8.
4.5. Adding Edge Detection in Preprocessing
This paper has carried out edge detection two times successively; the first time is to perform a wide range of edge detection extraction in the entire frame image. In the second, the edge detection is performed again after the lane detection after ROI selection. This detection further improves the accuracy of lane detection [22]. This section mainly performs the overall edge detection on the frame image, using the improved Canny edge detection algorithm. The concrete steps of Canny operator edge detection are as follows: First, we use a Gaussian filter to smooth the image (preprocessed image), and then we use the Sobel operator to calculate the gradient magnitude and direction. Next step is to suppress the nonmaximal value of the gradient amplitude. Finally, we need to use a double-threshold algorithm to detect and connect edges. Figure 9 shows the image after extraction with Canny edge detection.
4.6. ROI Selection
After edge detection by Canny edge detection, we can see that the obtained edge not only includes the required lane line edges, but also includes other unnecessary lanes and the edges of the surrounding fences. The way to remove these extra edges is to determine the visual area of a polygon and only leave the edge information of the visible area. The basis is that the camera is fixed relative to the car, and the relative position of the car with respect to the lane is also fixed, so that the lane is basically kept in a fixed area in the camera.
In order to lower image redundancy and reduce algorithm complexity, we can set an adaptive area of interest (ROI) on the image [19]. We only set the input image on the ROI area and this method can increase the speed and accuracy of the system. In this paper, we use the standard KITTI road database [26]. We divide the image of each frame in the running video of the vehicle into two parts, and one-half of the lower part of the image frame serves as the ROI area. Figure 10 shows the ROI selection of sample frames (a), (b), (c), and (d) which are processed by the proposed preprocessing. The images of the four different sample frames have been able to substantially display the lane information after being processed by the proposed preprocessing method, but not only the lane information but also a lot of nonlane noise is present in the upper half of the image. So we cut out the lower half of the image (one-half) as the ROI area.
4.7. Lane Detection
The lane detection module is mainly divided into lane edge detection and linear lane detection. This section implements the basic functions of lane detection and performs lane detection based on improved preprocessing and the proposed ROI selection.
4.8. Edge Detection
Feature extraction is very important for lane detection. There are many common methods used for edge detection, such as Canny transform, Sobel transform, and Laplacian transform [18, 24]. We have selected Canny transform which is better. As shown in Figure 11, we performed Canny edge detection after the proposed ROI selection.
4.9. Lane Detection
The methods of lane detection include feature based methods and model-based methods. The method based feature is used in this paper to detect the colour and edge features of lanes in order to improve the accuracy and efficiency of lane detection.
There are two methods to achieve straight lane detection. One is to use the Hough line detected function encapsulated by the OpenCV library commonly used for image processing, and draw lane lines in the corresponding area of the original image. The other is self-programming. In the header file, the ROI area is traversed to perform line detection for a specific range of angles [27].
Both methods can be reflected in the video, and the first method runs faster. Since this article focuses on the accuracy and efficiency of lane detection, we chose the first method (Hough line function in the OpenCV library) to run faster for linear detection. Moreover, because the Hough transform is insensitive to noise and can process straight lines well, Hough transform is used to extract lane line parameters in each frame of the image sequence for lane detection.
In image processing, the Hough transform is used to detect any shape that can be expressed in a mathematical formula, even if the shape is broken or somewhat distorted. Compared with other methods, the Hough transform can find noise reduction better than other methods. The classic Hough transform is often used to detect lines, circles, ellipses, etc. As shown in Figure 12, lane detection uses Hough of sample frames (a), (b), (c), and (d).
4.10. Lane Tracking Using Extended Kalman Filter
After completing the lane detection, the next step is to track the lane, which is also a key technology for smart and automated vehicle (SAV) [24].
Image edge detection technology and linear lane detection are technologies used to detect lane; then EKF is used to track these parameters one by one [22]. In this way, the tracking of lane lines is converted into the tracking of lane line parameters, which not only improves the tracking speed, but also introduces the method of Kalman tracking to improve the tracking accuracy.
The experimental results are shown in Figures 13 and 14. The real-time tracking lane line is detected in the video stream. Figure 13 shows different results of lane detection at different times (i), (ii), (iii), and (iv) in one video. Figure 14 shows different results of lane detection at different times (i), (ii), (iii), and (iv) in another video.
5. Results and Discussion
Figure 15 shows the preprocessing of four frames of images. Frame (a.i) and frame (b.i) are processed by basic preprocessing (without white feature extraction), and frame (a.ii) and frame (b.ii) are processed by the proposed preprocessing (with the white feature extraction). From the Figure 15 we can see that frame (a.ii) and frame (b.ii) which are processed by the proposed preprocessing can display the lane line. But there is a large amount of white residue in frame (a.i) and frame (b.i), and it is difficult to detect lane lines. Therefore, the basic preprocessing of the frame does not work well for lane detection. In view of these, we propose to add HSV colour conversion in the preprocessing stage and then extract the white features of the frame before the blurry ones, so as to achieve a better detection effect and improve the detection accuracy.
As shown in Figure 16, frame (a) and frame (b) are extracted white features, respectively.
Most research scholars directly perform ROI selection on the original image. In this paper, a new ROI selection method is proposed. Experiments show that the proposed ROI selection can improve the accuracy and efficiency of lane detection.
Figures 17 and 18 show the ROI selection of white feature. It can be seen from the figures that ROI selection of white feature cannot accurately detect the area of lane line, which will eventually produce a great error.
Half of the input frames are proposed as ROI selection. As shown in Figure 19, the ROI selection implemented on the original image is followed by edge detection and lane detection on the selected ROI area. Compared with Figure 12, the result of the final lane detection contains many nonlane areas, and the effect of the lane detection is poor. The more the lane parameters are marked, the less efficient the calculation is. Therefore, the proposed method in this paper can lower the number of lane parameters, thereby reducing the calculation time and improving the detection efficiency.
To quantify the accuracy of lane detection, we used the correct detection rate to evaluate the performance of our proposed method for lane detection under the data set used. For better results of the proposed method, we first set the size of the image in the data set to the same size and randomly take 300, 500, 800, 1000, and 1500 images as a test set in training sets. In order to verify the excellence of our proposed method, as shown in Figure 20, we compared the detection efficiency of the basic preprocessing method for lane detection with the detection efficiency of the proposed preprocessing method. Moreover, as shown in Figure 21, we also compare the lane detection efficiency of the lane detection method that selects the ROI area only based on the lane colour with the lane detection efficiency of the proposed ROI selection method. From Figures 20 and 21, we can see that the results of the proposed method achieve the highest correct detection rate to prove the effectiveness of our proposed method.
6. Conclusions
In this paper, we proposed a new lane detection preprocessing and ROI selection methods to design a lane detection system. The main idea is to add white extraction before the conventional basic preprocessing. Edge extraction has also been added during the preprocessing stage to improve lane detection accuracy. We also placed the ROI selection after the proposed preprocessing. Compared with selecting the ROI in the original image, it reduced the nonlane parameters and improved the accuracy of lane detection. Currently, we only use the Hough transform to detect straight lane and EKF to track lane and do not develop advanced lane detection methods. In the future, we will exploit a more advanced lane detection approach to improve the performance.
Data Availability
The proposed lane detection data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This paper was also supported by the project of Local Colleges and Universities Capacity Construction of Science and Technology Commission in Shanghai (no. 15590501300).