Abstract
The rapid development of the license plate recognition technology has made great progress for its widespread uses in intelligent transportation system (ITS). This paper has proposed a novel license plate detection and character recognition algorithm based on a combined feature extraction model and BPNN (Backpropagation Neural Network) which is adaptable in weak illumination and complicated backgrounds. Firstly, a preprocessing is first used to strengthen the contrast ratio of original car image. Secondly, the candidate regions of license plate are checked to verify the true plate, and the license plate image is located accurately by the integral projection method. Finally, a new feature extraction model is designed using three sets of features combination, training the feature vectors through BPNN to complete accurate recognition of the license plate characters. The experimental results with different license plate demonstrate effectiveness and efficiency of the proposed algorithm under various complex backgrounds. Compared with three traditional methods, the recognition accuracy of proposed algorithm has increased to 97.7% and the consuming time has decreased to 46.1ms.
1. Introduction
Vehicle plate licenses recognition (VPLR) plays a significant role in the field of intelligent transportation system. It has been widely used in traffic management, vehicle monitoring, suspect vehicle tracking, and many other fields. For example, in some cities in China, a new VPLR technology which enables drivers to pay parking fee using electronic wallet in a short time without leaving cars has received widely favourable reception. The parking fee from the drivers can be automatically collected by OCR (Optical Character Recognition) systems which can recognize license plates. However, errors sometimes occur when a vehicle is not identified or when a vehicle is wrongly identified as another vehicle. Therefore, RFID (Radio Frequency Identification) device and Bluetooth equipment can be combined to yield a better recognition performance [1–3]. Many new technologies like that are emerging and rising due to people’s changing demands; it is necessary to improve license plate location and recognition algorithms to increase vehicle management efficiency. In general, license plate recognition process consists of three steps: license plate localization, characters segmentation, and characters classification and recognition. Since the license plate localization is the first and essential step of the recognition process, the result has a direct impact on the accuracy of character segmentation and character recognition. However, the license plate can be easily affected by external factors such as lightning conditions, weather, and backgrounds; besides, most VPLR systems do not fully consider the complexity of background and illumination conditions in the practical application, so locating and detecting the license plate from original images accurately and efficiently are still vital steps and the main difficulties for successful license plate recognition [4–6].
At present, the representative license plate classification and recognition task is performed by machine learning methods, such as a Support Vector Machine (SVM) classifier, and the neural network methods including Backpropagation Neural Network (BPNN) [7–11]. In particularly, multilayer neural networks and backpropagation training is adequate for vehicle plate licenses reading and recognition [12–14]. In addition, since the high dimensional original images contain redundant information, it is better to extract useful image characteristics rather than using each pixel value of the images as feature vectors [15–17]. Therefore, the feature extraction method plays a significant role in these license plate recognition algorithms, which need to extract different sets of features as the input layer of the network.
In the past several years, different methods have been exploited to extract features to describe various characters by many researchers. In [18], Afeefa et al. use the Hierarchical Temporal Memory (HTM) spatial pooler to recognize the characters more accurately and efficiently. They present three algorithms to extract features but each of them has its limitations in recognizing some of the characters. In [19], it is found that the Position of Peaks algorithm can recognize English numbers successfully. However, it cannot distinguish between Arabic numbers 0, 1, and 6. In [20], a Pixel Density method is presented, which relies on processing pixels along vertical and horizontal lines taken across the character to achieve recognition. In [21], the proposed algorithm develops the line containing a feature or a set of combined features after the process of quantization, which can identify a specific character in the different training datasets. In [22], a hybrid license plate extraction method based on the edge statistics and morphology can detect the region of license plate quickly and accurately. In [23], a vertical traverse density (VTD) vector and horizontal traverse density (HTD) vector are proposed to describe each character object. This method is convenient to achieve recognition and it is less time-consuming; however, due to the similar structures in different characters, the proposed algorithm has difficulty distinguishing between T and L, Z and E, and some other groups. In summary, a robust character recognition algorithm based on an efficient feature extraction method further needs to be proposed to improve license plate recognition.
In order to overcome the problem that the features extracted from the characters cannot describe the detail clearly, which leads to error recognition, and that the number of features extracted is so large that its implementation costs much time, this paper proposes a robust license plate detection and character recognition algorithm based on a novel combined feature extraction model and BPNN. The combined feature extraction method uses VTD, HTD features, and edge distance features as the learning and training samples of the VPLR classifier. Compared with the traditional license plate detection and character recognition method, it has the following features and advantages:(1)A robust license plate detection and character recognition algorithm based on a novel combined feature extraction model and BPNN has been proposed in order to improve recognition efficient of VPLR algorithm.(2)The new combined feature extraction model consists of three sets of features: two sets of traverse density features (VTD, HTD) and a set of edge distance features, which contains more information for the neural network training.(3)The proposed algorithm can effectively determine and recognize variable license plates and has a good compatibility to regional difference under both weaker illumination and complex backgrounds.
Aside from this introductory section, the remainder of this paper is composed of 4 more sections. In Section 2 the license detection and character segmentation algorithm using preprocessing, accurately license plate positioning, and histogram of transformation numbers projection is proposed. Then, Section 3 describes the details of a new combined features extraction model with BPNN. Additionally, field test and experimental results analyses are given in Section 4. Finally, Section 5 concludes the paper with a summary of the results achieved in this paper and a discussion of future work.
2. License Plate Detection and Character Segmentation Algorithm
2.1. Preprocessing of Original Car Image
Considering actual conditions, there is much interference in original car images such as the size of the image, lighting, and imaging quality, which influence the recognition performance seriously. In order to locate the license plate quickly and accurately, preprocessing of original images needs to be carried out. Here, the original images are captured at a high resolution (1250×750), which ensures that both the small license plate and small characters on them can be processed and recognized using the proposed algorithms. The original image of a vehicle is shown in Figure 1.
In the first step, grey image conversion is used to transform color image to grey image. Then, considering that the grey level of license plate area is not obviously distinct from the other areas in the image, a grey level stretch processing is applied to enhance contrast between two parts. After that, a set of candidate regions are extracted after edge detection, image erosion operation, and morphological closed operation [24]. The whole steps of the original image preprocessing are described as follows:(1)First of all, original image is converted to grey image which decreases the computational cost.(2)Then, a grey level stretch processing is applied to enhance contrast between the license plate area and the other parts of image.(3)Edge is detected using Roberts operator to highlight the difference between the license plate frontier and the background.(4)Image erosion operation and morphological closed operation are performed, excluding the small parts which are certainly not parts of the license plate region.
The corresponding result of each step is shown in Figure 2.
(a) The grey image
(b) The grey level stretching image
(c) The edge detection image
(d) The morphological operation image
2.2. License Plate Coarse Extraction of the Candidate Regions
In the previous paragraph, it is shown that the candidate regions (marked with white color) of the license plate are clearly distinguished from the background (marked with black color). Then in the next step, the length and width information of the real license plate will be recorded to check the candidate regions one by one to select the true license plate area. As Chinese license plate outline size is 140mm440mm, the ratio is about 440/140≈3.15 [25], where setting the selected requirement is the aspect ratio ranging from 2 to 4. Using this method, the approximate poisoning of the license plate is achieved. The entire steps of license plate coarse extraction of the candidate regions are shown as follows:(1)Three candidate regions are marked with different colors.(2)According to prior experience and information mentioned above, the most likely area of the license plate region is verified which is marked with white color while the other candidate regions are all removed from the image.(3)According to the location of the license plate region, the corresponding position in the grey-scale image is located.
Figure 3 illustrates the result of each step in the processing.
(a) Marked image
(b) Selected license plate region
(c) The license plate region detection in the original grey image
2.3. Accurate Positioning Based on Horizontal and Vertical Integral Projection Method
In order to segment and recognize the character accurately, it is necessary to process the approximately positioning of the license plate to get the accurate positioning of the license plate. In this paper, the horizontal integral projection and vertical integral projection are adopted [26]. A horizontal first order difference is carried on coarse location image to achieve the horizontal accurate positioning image . The horizontal integral projection can be described by the following equation:where, is the pixel value of the image , is the pixel value of the image , and and are the height and the width of the image , respectively. Then the projection value of the row named can be obtained by accumulating the pixel value of image per row, which can be depicted as follows.
Considering that the edge of the image is not smooth enough, the average value is used as to locate the boundary efficiently. The equation can be expressed as follows.
The horizontal integral projection of the coarse location image is shown in Figure 4.
After this processing, the statistics of each row are saved in array, where is the value of the corresponding row. Analysing horizontal projection value, character regions are generally corresponding to the intermediate dense and high regions which accounted for position, and the upper and lower edge of license plate are corresponding to the left and the right regions which accounted for position. In this way, the license plate can be located accurately in vertical direction. Here, the algorithm is described as follows:(a)according to experimental results, setting the 0.6 of the maximum value of as threshold;(b)from left to right, researching the array to find the first element value which is bigger than threshold, taking down the index of the value as ;(c)from right to left, researching the array to find the first element value which is bigger than threshold, taking down the index of the value as ;(d)extracting the image between the row and row as the accurate poison in vertical direction.
The difference between the coarse image and processing image is shown in Figure 5; it is shown that the two screws have been removed, while the license plate frames in horizontal direction still exist.
(a) Coarse image
(b) Accurate poisoning image in vertical direction
Similarly, the vertical integral projection is applied to locate the license plate accurately in horizontal direction. The equation can be depicted as follows:where is the pixel value of the image which has been processed above, is the pixel value of the image , and and are the height and the width of the image , respectively. Then the projection value of the column named can be obtained by accumulating the pixel value of image per column, which can be depicted as follows.
Considering that the edge of the image is not smooth enough, the average value is used as to search for the boundary efficiently. The equation can be expressed as follows.
The vertical integral projection of the processed image is shown in Figure 6.
Here, the statistics of each column are saved in array, where is the value of the corresponding column. From Figure 6, we can conclude that characters regions are generally corresponding to the peaks and the valleys while some small parts in the left and right are corresponding to license plate frames. Then we can use the algorithm to locate the license plate accurately in horizontal direction which can be described as follows:(a)according to experimental results, setting the 0.7 of the average value of as threshold;(b)from left to right, researching the array to find the first element value which is bigger than threshold, taking down the index of the value as ;(c)from right to left, researching the array to find the first element value which is bigger than threshold, taking down the index of the value as ;(d)extracting the image between the column and column as the accurate position in horizontal direction.
The final accurate poisoning image is shown in Figure 7; it can be seen that the image contains exactly the seven characters without any other area.
2.4. Character Segmentation Method Using the Number of Alternating White and Black
After extracting the accurate positioning image of the license plate, the character segmentation is researched at this stage. The license plate image consists of seven characters, a dot, and the space between them. To get the images only containing each character, it is natural to seek for the starting and ending points of each character, so an algorithm based on calculating the numbers of white and black transformations per column is presented. The step of the algorithm is described as follows:(a)Grey-scale image is converted to binary one by Otsu method [27].(b)An array is created to store the transformation value of each column, where is the index of each column and is the width of the license plate image. The initial values are 0.(c)Each pixel value is searched per column from up to bottom; if the pixel value changes from 1 to 0 or 0 to 1, it is defined as a transformation, and then one is added to the corresponding value of the column ; otherwise, the original value stays.(d)Draw the histogram of until the last column is completely processed.
As shown in Figure 8, eight simply connected regions stand for seven characters and a dot, respectively. The blank space between each region stands for the interval between two adjacent characters on the license plate. Therefore, the first character starts with point 1 (the left edge) and ends with point A marked in Figure 8. Thus, the first character can be separated ranging from column 1 to column A from the image. The second character starts with point B and ends with point C. Thus, the second character can be separated ranging from the column B to the column C from the image. Noting that the point D represents the dot which needs to be removed, the remaining image can start with the ending point of the dot which is about 2/3 length distance of B to C from the point C. In this way, the remaining five characters can be segmented one after another. The final segmentation result is shown in Figure 9, which has also been introduced in [28].
3. Character Recognition Based on Traverse Density Features with Edge Distance Features
3.1. Features Extraction Algorithm Based on VTD and HTD Methods
The traditional traverse dense features consist of horizontal traverse density (HTD) and vertical traverse density (VTD), which has been explained in [28]. Here, in order to describe procedure of the HTD features extraction clearly, a character object “N” is used.
The character N is selected after the binary operation. Scanning the column and row of the image pixels one by one and recording the number of changes in black and white pixels, the rule is shown as follows:where the initial feature value is 0 and is the pixel value of the image. In the case of the column scan of the character “N”, from left to right, the first column is alternating between black and white once, so the corresponding value is marked with one small black dot in the box. It can be clearly seen from Figure 10(a) that the most number of alternating black and white pixels is once [28]. These characteristic values are accurately remarked in the box. From left to right, a series of feature values are obtained, which can be recorded as vertical traverse density (VTD). In the same way, the character object “N” will be scanned from top to bottom, the side parts of the image have altered form black to white twice, and the intermediate part of that is three times. A set of feature values will be recorded as the horizontal traverse density (HTD). The VTD and HTD are features used for character recognition, and they illustrate the image features of the target characters accurately.
(a) The image of character “N”
(b) The HTD features of character “N”
(c) The VTD features of character “N”
As is shown in Figure 10, the HTD and VTD feature vectors of character “N” are “22222222222333333333333322222222222” and “11111111112211111111”, respectively. In this method, the feature vectors of all the letters and numbers can be obtained. Since the VTD and HTD feature extraction is based on scanning each column and row of the image, the dimensions of the VTD and HTD feature are the width and height of the character to be measured. It is known to us that the larger the size of target image is, the more useful information it contains. As a result, the recognition rate will be higher in theory while it costs more time in manipulation, so it is important to make a balance between the recognition rate and the computation time of the algorithm. According to the experimental results, the final optimal license plate size is 35×20, i.e., the dimensions of the two feature vectors are 35 and 20, respectively. To improve the recognition rate and the efficiency of the system, all the character images to be measured and the template character images are normalized to size 35×20.
3.2. Improved Features Extraction Algorithm Using Edge Distance Features
The algorithm mentioned above uses VTD and HTD features as two sets of characteristic vectors. However, when handling some words which have similar traverse density structure like the character “E” and the character “Z”, the character “T” and the character “L”, it does have the same feature vectors which may cause misleading results in recognition.
As can be seen from Figure 11, it is clear that both the character “E” and the character “Z” have the same VTD and HTD feature vectors; hence characters “E” and “Z” are identical in feature vectors which are indistinguishable [28]. In order to address this problem, another set of features is put forward to compensate for the deficiency in traditional feature extraction algorithm.
Here, the proposed feature extraction method focuses on the edge distance features. The step of extracting the edge distance features can be described as follows:(a)From the up to the bottom, search each pixel of row from the left to the right of the character image.(b)Find the first pixel of the row whose value is 1 (white), recording the index of the pixel, setting it as , where and is the height of the character image.(c)The elements in the array consist of the feature vector.
The algorithm can be seen in Figure 12. In this way, the edge distance feature values of the character “Z” are changing from 1 to , where is the width of the image, while the most edge distance feature values of the character “E” are near to 1, so the difference of the feature vector between two characters is obvious when adding this set of features [28].
Thus, three different sets of features have been defined in the improved features extraction algorithm: the vertical traverse density, the horizontal traverse density, and the distance from the left edge to the first white pixel.
3.3. License Plate Characters Classification and Recognition Using Backpropagation Neural Network
In order to train a good recognition performance model, the license plate characters of the training samples must be collected. Our goal is to recognize the Chinese characters, the numbers 0-9, and the English letters A-Z of the license plate. The samples of the license plate character images are stored by different classification in the following folder according to the corresponding correct character information. For instance, the collection of license plate training samples of Chinese character “苏”, English letter “E”, and the number “6” is, respectively, shown in Figures 13(a), 13(b), and 13(c). They stand for one representation of the Chinese characters, the English letters, and the Arabic numbers, respectively.
(a) Chinese character
(b) English letter
(c) Arabic number
The proposed character recognition method based on traverse density features with edge distance features has good real-time performance and the algorithm is convenient to implement. If it is combined with neural network, the recognition accuracy, especially the similar characters recognition accuracy, will be further improved by utilizing the specific learning process of the neural network. Here, BPNN is chosen as a training model [29–31]. As is researched in the previous paragraph, three sets of feature vectors can describe most of the characteristics of the target characters, and these three sets of feature vectors can be combined as a vector used as input vector for the neural network designing and building. Structure of the letters and numbers mixed classifier based on BPNN is shown in Figure 14.(1)The number of neuron nodes in the input layer: The neuron nodes in the input layer are determined by the dimension of the features extracted from the images. The size of license plate character images used in this article is 35×20. That is to say, the dimension of the HTD features of the character is 35, the dimension of the VTD features of the character is 20, and the dimension of the edge distance features is 35. Connecting three sets of features into one feature vector, a 90-dimensional vector can be obtained as the input vector of the neural network. Therefore, the input layer has 90 nodes.(2)The number of neuron nodes in the output layer: The neuron nodes in the output layer are decided by the total numbers of Chinese characters, English letters, and digital numbers waiting to be classified. There are 26 English letters, 10 digital numbers, and 36 alphanumeric characters in the system, so the output layers have 26 nodes, 10 nodes, and 36 nodes corresponding to the English letters classifiers, digital numbers classifiers, and alphanumeric characters classifiers, respectively.(3)The number of neuron nodes in the hidden layer: The nodes numbers of hidden layer are determined by many factors, having a great impact on the performance of the training model. According to the experimental results and empirical formula, the nodes number of English letters classifier is 48, the nodes number of digital number classifier is 27, and the nodes number of letters and digital numbers mixed classifier is 64.
To train the neural network effectively, there are 100 vehicle license plate character images used in the experiment, of which 80 images are randomly selected as training samples, and the remaining 20 images are selected as test samples. The mean squared error of the training processing of letters and numbers mixed classifier is plotted in Figure 15.
4. Experiment and Analysis
In order to test and verify performance of the proposed algorithm, field tests are provided in this section. To demonstrate the proposed license plate detection algorithm, several experiments using samples with different backgrounds have been conducted. Figures 16(a) and 16(b) show the real license plate detection, segmentation, and recognition results of two small family vehicles. As can be seen from Figure 16, the proposed license plate detection and recognition algorithm can effectively determine license plate location and complete recognition function under complex backgrounds.
(a) A small brown family car
(b) A small white family car
Besides, to better present the test results, a vehicle license plate recognition system has been designed. The entire processing results of the original image under more complex and weaker illumination backgrounds are shown in Figure 17. As illustrated in Figure 17, the proposed license plate detection and recognition algorithm can detect and recognize the license plate under the weak ambient illumination since image under the grey level stretch processing is robust to the illumination information.
(a) A SUV black family car
(b) A SUV silver family car
Moreover, 100 license plates are collected in different backgrounds and different illumination condition which are used to test license plate detection and characters recognition. As a result, 95 license plates are completely correctly identified. The correct license plate recognition rate is 95 percent. There are total 700 valid characters which have been measured and 684 of them are correctly identified, which shows that the character recognition rate is 97.7 percent. The average recognition time of 100 license plates samples for single license plate is 707ms. Besides, the recognition rate of letters and numbers and the consuming time of them for single character are summarized and shown in Table 1, whose analysis is also given in [28].
Then, to evaluate the proposed features extraction algorithm, 100 license plates, including 50 percent samples collected by the authors and another 50 percent samples collected from some vehicle libraries, are used to compare the accuracy rate and run time of the several introduced methods. Compared with the conventional methods, the proposed algorithm occupies a dominant position in the recognition rate and the computation time. Experimental results are summarized in Table 2. The proposed algorithm shows a precision of 97.7% and consuming time of 46.1ms, outperforming the three conventional methods.
Finally, in order to further verify the adaptivity and compatibility of the proposed algorithm, tests are also implemented using Hong Kong vehicle license plate. Hong Kong license plate is made up of English letters and Arabic numbers without Chinese characters. Due to the difference in size, module, and font between Chinese mainland license plate and Hong Kong SAR license plate, it is still challenging to the presented plate detection and feature extraction method. The license plate detection, segmentation, and recognition results of several Hong Kong vehicle license plates are shown in Figure 18. As can be seen from Figure 18, the algorithm can accurately detect and recognize the Hong Kong license plate, which indicates that the proposed algorithm has a good compatibility to regional difference in license plate.
(a) A pale blue family car in Hong Kong
(b) A grey family car in Hong Kong
(c) A white family car in Hong Kong
5. Conclusion
This paper presents a robust license plate detection and character recognition algorithm based on a combined feature extraction model and BPNN. The experimental results demonstrate that the proposed algorithm obtains a good recognition performance and compatibility in identifying different license plates under weaker illumination and complex backgrounds. Compared with the three traditional methods, the proposed method provides a better recognition accuracy and time-consuming performance with 97.7% and 46.1ms. In the future work, extending the VPLR system would be further considered with RFID device and Bluetooth equipment combination to improve recognition accuracy and fit more complex requirements.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work is partially supported by the National Natural Science Foundation of China (Grant No. 61601228, 61803208, 61403210), the Natural Science Foundation of Jiangsu (BK20161021, BK20180726), National Key Research and Development Program of China (Grant No. 2017YFB1103200), and the Natural Science Foundation of Jiangsu Higher Education Institution (17KJB510031, 18KJB120005) and sponsored by the Open Program of Jiangsu Key Laboratory of 3D Printing Equipment and Manufacturing, Item number 3DL201607.