Abstract
Industrial Internet of Things is the core field of smart city. And intelligent detection is an important application field of industrial Internet of Things. Demand of the industrial is particularly urgent. In particular, the defect detection of mobile phone shells (MPS) has always been a common problem for famous mobile phone companies. A compression-free defect detection method (CFDDM) for MPS based on machine vision is proposed in this paper. Firstly, affine transformation is utilized to solve the angle deviation of MPS in different images. Then, edge detection, binarization, and open operation are combined to highlight the edge region based on the results of angle adjustment. It is convenient for region of interest (ROI) extraction and clipping. Finally, the method of gray histogram contrasting is utilized for defect detection according to the results of ROI clipping. And the detection results are obtained. In this paper, MPS data set is utilized for many tests. The results show that the proposed method can effectively detect whether there are defects in MPS data set without image compression. The recognition accuracy is 100%. The recognition time of a single image is about 4.56 s, which is better than other defect detection methods.
1. Introduction
The core goal of the construction of industrial Internet of Things is to build an industry wide economic operation monitoring system for the smart cities [1]. It can be monitored, quantified, predictable, and intelligent decision-making. Intelligent detection is the core application field of industrial Internet of Things. It is not only the core content of intelligent transformation of industrial production line, but also the related detection algorithm is widely utilized in many aspects of smart city, such as logistics management and commercial application.
As an indispensable component of smart phones, the mobile phone shell (MPS) plays a role in protecting the internal components of mobile phones, such as circuit boards and batteries. In addition, the functions of beautifying mobile phones and highlighting features are more popular with consumers. Therefore, the detection of MPS in industrial field is becoming more important. Most of the traditional detection methods of MPS are manual defect detection. The average time of defect detection manually is 15 s [2]. At the same time, the accuracy of manual detection depends on the detection experience and proficiency. It increases the risk of false detection. A method that is standardized, time-saving, and accurate detection is pursued by researchers based on such problems.
A compression-free defect detection method (CFDDM) is proposed in this paper for saving above problems. Firstly, the angle of MPS in the image is adjusted by affine transformation, so that MPS of each sample image is perpendicular to horizontal line. Then, the sample image is grayed. And the gradient image is obtained by Sobel operator. A low-pass filter [3] for filtering denoising is utilized in this paper to eliminate the influence of noise. Then, the image is binarized and processed by the open operation [4]. The purpose of this step is to highlight the outline of the MPS as much as possible. Then, the detect ROI of MPS is processed by OpenCV. Finally, ROI is cut to get MPS image without background. The gray histogram of each region of interest (ROI) image is extracted as features in the feature extraction part [5]. Firstly, a gray histogram is extracted that belongs to ROI of defect-free sample and ROI of the sample to be tested. Then, the system obtains the detection results by comparing the two gray histograms in the decision-making part.
The rest of this paper is structured as follows. In Section 2, the related work of industrial inspection method and system based on machine vision is reviewed. Section 3 describes the defect detection method of MPS image without compression. Section 4 gives the results of experiment and inspection. In addition, the results of various defect detection methods are compared in Section 4. Finally, the conclusion of the detection system is given in Section 5.
2. Related Work
The detection method of MPS began to receive more attention by researchers as an indispensable key part of smart phones. Defect detection systems have been widely utilized in the field of manufacturing with breakthrough and innovation in the field of industrial machine vision [6]. It was based on machine vision for industrial products.
2.1. Feature-Based Defect Detection System
Wang et al. [7] proposed a product defect detection system to extract ROI and an inverted residual cascade network to extract features. It was based on Hough Transform (HT). In this system, HT is utilized to extract ROI of the product under the cooperation of a fixed-point rectangular light source. It greatly reduces the problem of too long detection time caused by calculating unrelated background by computer. Aminzadeh and Kurfess [8] proposed a new feature extraction method. The histogram of background and defect regions are compared to find an optimal threshold. Then, the optimal threshold is utilized for threshold segmentation. After that, feature points or regions are extracted from the binary images after threshold segmentation. Finally, a machine learning or deep learning model is established according to the extracted features to obtain the detection results.
2.2. Defect Detection System Based on Template Matching
The defect detection system based on template matching can usually get the difference between the template image and the image to be tested [9]. Then, the system makes a judgment whether there are defects in the image to be tested [10]. This method has many advantages, such as high efficiency and low error. It played an important role in the field of industrial detection and industrial identification [11]. Jian et al. [12] proposed a mobile phone screen defect detection method. It combines contour-based image registration, image subtraction, and gradient projection. At the same time, an improved fuzzy c-means clustering (IFCM) was proposed for defect classification. Tsai and Hsieh [13] proposed a new template registration method. Firstly, the expected maximum algorithm is utilized to assign weight values to each edge point of the image and the template to be tested. Then, the edge points corresponding to the weight are found in the image to be measured and template. Finally, the alignment between the image to be tested and the template is realized.
2.3. Defect Detection System Based on Deep Learning Neural Network
The defect detection system based on deep learning neural network is quite different from that based on machine learning. The defect detection system based on deep learning needs to be inputted a large number of data sets to improve recognition accuracy. Improvement of accuracy is by self-learning of neural network. Zhang et al. [14] proposed a classifier for defect image detection based on CNN. Ho et al. [15] proposed an automatic optical inspection system based on CNN. It is utilized to detect five kinds of defects in mold manufacturing process. Jisu et al. [16] proposed a weighted multi-DLNN neural network for glass defect detection and classification. It is based on DLNN neural network. The new neural network proposed can utilize multichannel data to achieve high accuracy effectively.
However, the defect detection method based on deep learning has serious shortcomings. The accuracy and practicability of network depend on the quantity and quality of training samples heavily [17]. Moreover, the acquisition of data sets is one of the problems for researchers. In this paper, CFDDM for MPS image based on machine vision is proposed. It can achieve 100% recognition accuracy without a large number of data training and image compression.
3. Detect Defection Method for MPS
CFDDM proposed in this paper consists of five parts: image acquisition, image preprocessing, ROI extraction, image feature extraction, and defect recognition.
3.1. Image Acquisition
Core of CFDDM proposed in this paper includes image acquisition and image processing.
The light source of this experiment is a strip light source that is composed of a high-density LED array. It is suitable for surface size detection and edge defect detection [18]. Therefore, this experiment utilizes a strip light source.
In this experiment, an 8 K industrial camera is utilized in this experiment. The resolution of MPS captured by the 8 K industrial camera is 81928000. The experimental platform of image acquisition is shown in Figure 1. It needs to place the samples on the platform before image acquisition. Then, strip light source is turned on to reduce the interference of ambient light. It can capture clearer MPS images. The pulley under the platform can be regarded as the -axis of the platform. Moving the pulley back and forth until MPS appears in the field of vision of the 8 K industrial camera. The camera also moves with the frame in -axis direction to ensure focus correctly. The captured image is a gray image with a resolution of 81928000 and BMP format.
The specific models of the camera, lens, and light source are shown in Tables 1–3. The samples of defective and nondefective MPS utilized in the experiment are shown in Figures 2(a) and 2(b).
(a)
(b)
3.2. Image Proprogress
3.2.1. Image Denoising
Image denoising is utilized in the experiment to reduce the impact of noise points in the image. Gaussian low-pass filter is utilized to denoise the MPS sample image in this paper. The one-dimensional transfer function of Gaussian low-pass filter and the two-dimensional transfer function are obtained as follows:
where denotes the standard deviation. and denote the coordinates of pixels in image. Gaussian low-pass filter is utilized for digital image processing in this paper. Therefore, two-dimensional information should be processed by computer. Two-dimensional transfer function is utilized in the paper.
3.2.2. Image Graying
Data set in this experiment contains three channel images. Graying is utilized to reduce the number of channels in order to reduce the amount of image computation [19]. The optional graying methods are as follows: maximum value method, average value method, and weighted average method. The principles of the three methods are different. The calculation equations are as follows: where denotes the pixel value of the channel at the position of the pixel . denotes the pixel value of the channel at the image position. denotes the pixel value of channel at the image position.
3.3. ROI Extraction
There are not only ROIs but also a large area of useless background area in the images that taken by industrial cameras [20]. These regions not only increase the detection time of the defect detection system but also interfere with the recognition results. The following image processing method is utilized in this paper to get ROI to reduce interference.
3.3.1. Affine Transformation
Affine transformation is utilized to adjust ROI angle of MPS [21]. Therefore, ROI is perpendicular to horizontal line. MPS before affine transformation is shown in Figure 3(a), and the processed image is shown in Figure 3(b).
(a)
(b)
Affine transformation is suitable for image translation, rotation, scaling, and mirror operation. The rotation operation of affine transformation is utilized mainly in this paper. The matrix form of the operation is as follows:
where denotes coordinates of the pixel before transformation. denotes coordinates of the pixel after affine transformation. denotes the angle of image rotation. denotes the translation amount of pixels.
3.3.2. Calculate Gradient of the Image
ROI was extracted in this experiment. Features of the image in ROI can be reflected well by gradient of the image.
Sobel operator [22], Prewitt operator [23], and Laplace operator [24] are utilized in image gradient computation commonly.
Sobel operator is a first derivative discrete differential operator. Convolution equations in direction and direction are as follows:
where denotes the image to be acted on. Convolution results of sample image direction in this experiment are shown in Figures 4(a) and 4(b).
(a)
(b)
(c)
Firstly, the gradient values in the and directions of each pixel of image are calculated. Then, the approximate gradient value of pixel is calculated by equation (5). It is shown as follows:
Sobel operator is utilized to process the sample image in this experiment. The gradient result is shown in Figure 4(c).
Prewitt operator is a first derivative discrete differential operator like Sobel operator. Prewitt operator calculates the image gradient by averaging firstly and then calculating the difference. Equations for calculating the gradient in direction and direction are as follows:
where denotes the image to be acted on. The and directions of the gradients obtained by Prewitt operator are shown in Figures 5(a) and 5(b) in this experiment.
(a)
(b)
(c)
Prewitt operator is utilized to process the sample image in this experiment. The gradient result is shown in Figure 5(c). Laplace operator is a second order differential linear operator. It is defined as follows:
The second derivative can detect the change of image gradient by calculating the change of pixel intensity in the direction of image pixel. The difference partial derivative equation of Laplace operator in and directions is shown as follows:
Laplace operator is utilized to process the sample image in this experiment. The gradient result is shown in Figure 6.
It can be seen that MPS contours of the sample image can be detected by each operator from the results of the three gradient operators. However, the detection features of the three gradient operators are different. It can be observed from Figure 4(c) that the edge of MPS can be detected completely by Sobel operator. And the contrast between edge pixels and background pixel in the binary image is high. It can be observed from Figure 5(c) that the edge of MPS can be detected completely by Prewitt operator. And the contrast is low that between edge pixel and background pixel in the binary image. The contrast of edge and background detected by Laplace operator is between Sobel operator and Prewitt operator. The result of contrast is shown in Figure 6. They show that Sobel operator has the best detection effect. It can detect the contour of the pixel area of the MPS. The contrast is higher than other operators. It makes subsequent binarization methods better.
3.3.3. Binarization
ROI is extracted by this experiment. The feature of the image in ROI can be reflected well by the gradient of image. The contrast of MPS edge image pixel obtained by Sobel operator is far from enough. It leads to the complex gray value in the image. It also brings difficulties to the subsequent edge contour recognition process. Therefore, binarization is utilized to enhance the contrast of MPS edge contour. It is helpful for morphological operation to filter fine pixels and expand the image edges. The result of binarization is shown in Figure 7(a).
(a)
(b)
(c)
3.3.4. Morphological Open Operation
Morphological operation is utilized to fill the small holes of the image in this experiment. It can enhance the integrity of the image and facilitate following algorithm for edge extraction. Binary image is taken morphological open operation 4 times. The result is shown in Figure 7(b).
It can be seen from the result figure that the open operation can reduce the black area in MPS appropriately. It has a certain noise reduction effect and improves integrity of MPS images. It is conducive to ROI extraction.
3.3.5. Edge Tracking Algorithm Based on Topology
Edge tracking algorithm based on topology structure [25] is utilized to recognize ROI from the image in this experiment. The recognition result is shown in Figure 7(c).
This algorithm traverses the pixel of an image to determine whether the pixel is in the edge region. The judgment conditions are as follows: where denotes the position of the pixel in the image. denotes the pixel value of the pixel in the position.
The contour obtained by the edge tracking algorithm based on topology is not a rectangular region of ROI. It is an irregular outer contour region. This makes it impossible to cut edges. Therefore, the minimum circumscribed rectangle is marked in this experiment according to the irregular external contour. ROI is cut by the software according to the minimum circumscribed rectangle. The calculation process of the circumscribed rectangle is as follows:
The minimum circumscribed rectangular area is shown in Figure 7(c). The red line mark is utilized in this experiment to observe the detection results conveniently.
It can be seen from Figure 7(c) that the result of ROI annotation is satisfactory. It shows that the irregular edge of ROI is detected successfully by the method. It also obtains a relatively complete minimum circumscribed rectangle of ROI. In addition, it shows that the image processing method utilized in this paper is effective. Finally, ROI is cut according to the position information of the four corners of the minimum circumscribed rectangle. The clipping result is shown in Figure 8.
It can be seen from Figure 8 that cutting effect is satisfactory. The unrelated regions are cut to the maximum extent on the basis of retaining ROI.
3.4. Features of Image Extraction
Gray histogram of images is extracted as a feature in this paper. The gradient histogram of MPS sample image is shown in Figure 9.
Gray histogram denotes the number of gray levels in an image. -axis denotes the value of the pixel range from 0 to 255. -axis denotes the statistics of the number of a certain gray level in the image. The longer blue line in the image, the more times gray pixels appear in the image. The calculation equation of gray histogram is as follows:
Among them, denotes the gray level of a pixel in image. denotes the probability of the occurrence of a pixel with a pixel level of in whole image. denotes the number of pixels with a pixel level of in the whole image. denotes the total number of pixels in the image.
3.4.1. Image Defect Detection
Difference of content between images can be reflected as the difference of the pixel value of each pixel in image. In this paper, it is proposed to judge whether the sample to be tested is defective. It is by comparing the gray histograms of MPS nondefective and MPS to be tested. A difference threshold is set before comparison. MPS of the current image is defined as a defective product, if the difference between two gray histograms is greater than the difference threshold. MPS is defined as defect free product, if the difference between two gray histograms is less than the difference threshold. The difference threshold in this paper is set at 0.98.
4. Model Evaluation
4.1. Experiment Preparation
In this section, CFDDM is tested and evaluated. The data set in this experiment contains a total of 200 MPS images including defective samples and nondefective samples. The number of defective MPS sample images is 150. It accounts for 3/4 of the total. The number of defect free MPS sample images is 50. It accounts for 1/4 of the total. The resolution of each sample image is 81928000.
The definition of accuracy of CFDDM proposed in this paper is shown as follows:
where denotes the number of correct test results. denotes the number of wrong test results. denotes the number of defective images identified as defective correctly. denotes the number of nondefective images identified as nondefective correctly. denotes the number of nondefective images identified as defective wrongly. denotes the number of defective images identified as nondefective wrongly.
This experiment is based on machine vision OpenCV [26], using PyCharm platform for image processing and programming (a personal computer (CPU: Inter(R) Core(TM) i5-6200U; GPU: AMD Radeon(TM) R5 M430; RAM: 4GB).
4.2. Experiment Results
Firstly, the data set is processed by the method proposed in this paper. Then, 200 MPS images to be tested and 1 nondefective sample image were detected by gray histogram comparison. 50 samples of nondefects were identified as nondefective successfully. 150 samples with defects were identified as defective successfully. The accuracy was 100%, and time consuming was 4.56 s.
Excellent accuracy due to the following aspects is as follows: (1)Affine transformation is utilized to adjust the angle of MPS. It is beneficial to cut the unrelated background region in the stage of ROI extraction(2)The influence of noise from the background region is reduced by ROI extraction(3)Binarization and open operation are utilized to process the edge image in this experiment. Large edge pixel regions are combined to improve the integrity of MPS edge region. It also improves the detection effect of the minimum rectangular region of ROI
The method of MPS surface defect detection based on machine vision is composed of image preprocessing stage, ROI extraction stage, feature acquisition stage, and defect recognition stage in this paper. The time consumption of each part is shown in Table 4. Total time is 4.56 s. Time proportion of each part is shown in Figure 10.
4.3. Identification Performance Comparison
Edge extraction and ROI extraction are significance in this experiment. The effect of ROI extraction is affected by the results of edge extraction seriously. Sobel operator is utilized for edge extraction in this paper. The size of the convolution kernel has important influence on the result of edge extraction. It affects the result of ROI extraction directly.
Time consumption of ROI extraction by utilizing different sizes of Sobel operator is shown in Figure 11. The result of ROI extraction by utilizing different sizes of Sobel operator is shown in Figure 11.
(a)
(b)
(c)
Three different colored lines in Figure 12 denote Sobel operators with different convolution kernel sizes. convolution kernel utilized in this paper is denoted by a black line. Red line denotes convolution kernel. Blue line denotes convolution kernel. -axis denotes the time consumption of three kinds of Sobel operators to extract ROI. Time consumption of Sobel edge extraction was recorded 5 times in this experiment. The number of experiments is shown in -axis. The time of each work is shown in -axis. It can be seen from Figure 12 that ROI extraction process by utilizing convolution kernel takes the least time. It is 1.92 s at most. The convolution kernel of takes the most time. It is 2.74 s at most.
It can be seen from Figure 11 that ROI can be detected completely by Sobel operator. Sobel operator misjudges a small region of MPS as ROI in the process of ROI recognition. Wrong recognition result leads to the failure of ROI clipping directly. A small region of MPS is misjudged as ROI by Sobel operator in the process of ROI identification. Wrong recognition result leads to the failure of ROI clipping directly. ROI does not to be recognized in the correct region by Sobel operator. However, the rectangular image contour is recognized as ROI in the process of ROI identification. It makes impossible to cut ROI. To sum up, the Sobel operator utilized in this paper is the best.
More deep learning networks are utilized to detect product defects in the field of industrial product inspection. In this part, CFDDM proposed in this paper is compared with Alexnet, LSTM, and CNN. It is worth noting that the data sets utilized by all detection methods are MPS images after preprocessing. The accuracy of the comparison experiment is shown in Figure 13.
-axis denotes the type of defect detection method. -axis denotes accuracy. It can be seen from the Figure 13 that the accuracy of CFDDM proposed in this paper is 100%. It is higher than other neural networks.
5. Conclusion
It is inevitable to produce defective products due to improper operation of work in the processing of MPS. At present, most factories adopt manual observation to identify defects. The method is inefficient. CFDDM based on machine vision is proposed in this paper. It is utilized for the defect detection of MPS. CFDDM does not need a large number of data sets of training model. It also does not need to compress image after comparing with the conventional deep learning methods. It only needs to process image several times to get the defect recognition results. The test results show that the detection time is about 4.56 s. Accuracy can reach 100%. In the future work, we will explore the implementation of MPS defect classification on the basis of this study.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was supported by the Science and Technology Program of Guangzhou (No. 2019050001), project of Shenzhen Science and Technology Innovation Committee (JCYJ20190809145407809), project of Shenzhen Institute of Information Technology School-level Innovative Scientific Research Team (TD2020E001), Program for Guangdong Innovative and Entrepreneurial Teams (No. 2019BT02C241), Program for Chang Jiang Scholars and Innovative Research Teams in Universities (No. IRT_17R40), Guangdong Provincial Key Laboratory of Optical Information Materials and Technology (No. 2017B030301007), Guangzhou Key Laboratory of Electronic Paper Displays Materials and Devices (201705030007), and the 111 Project.