Abstract
Online defect detection system is a necessary technical measure and important means for large-scale industrial printing production. It is effective to reduce artificial detection fatigue and improve the accuracy and stability of industry printing line. However, the existing defect detection algorithms are mainly developed based on high-quality database and it is difficult to detect the defects on low-quality printing images. In this paper, we propose a new multi-edge feature fusion algorithm which is effective in solving this problem. Firstly, according to the characteristics of sheet-fed printing system, a new printing image database is established; compared with the existing databases, it has larger translation, deformation, and uneven illumination variation. These interferences make defect detection become more challenging. Then, SIFT feature is employed to register the database. In order to reduce the number of false detections which are caused by the position, deformation, and brightness deviation between the detected image and reference image, multi-edge feature fusion algorithm is proposed to overcome the effects of these disturbances. Lastly, the experimental results of mAP (92.65%) and recall (96.29%) verify the effectiveness of the proposed method which can effectively detect defects in low-quality printing database. The proposed research results can improve the adaptability of visual inspection system on a variety of different printing platforms. It is better to control the printing process and further reduce the number of operators.
1. Introduction
The online defect detection system based on machine vision is widely used in the field of industrial automation, such as welding defect detection [1], glass manufacturing industry [2], machine-parts processing [3], printed circuit board industry [4], textile industry [5], and printing industry [6, 7]. In printing industry, manual detection has been far from meeting the quality control requirements of modern large-scale printing production. Online defect detection system is an indispensable link to ensure the quality of printed matter [8]. Figure 1 gives an example of industrial printing line, and Figure 2 shows a typical defect detection system using machine vision for roll-to-roll printing line.
This paper is organized as follows. We present related research in Section 2, and Section 3 gives the methodology, which includes the architecture of the proposed defect detection system, image registration method, and feature extraction method. The experimental results and discussion are given in Section 4. Section 5 presents the conclusion and future works.
2. Related Research
Many defect detection methods have been proposed for roll-to-roll printing process, and these methods can effectively detect a variety of printing defects in real time. Paper [9] detected some common printing defects, such as ink drop, stripe, character loss, and color defect. Compared with the traditional method to extract the gradient edge of gray image, the proposed edge detection algorithm has better detection performance, and it can reduce the information loss of RGB three channels and make the edge extraction more accurate. Paper [10] provided an image fusion method, which used multi-channel image subtraction to segment defects. The method can update the reference image continuously with the printing production. Next, a twice template matching algorithm was established in paper [11], which firstly matched the template and then performed differential operation on matched image to find the location of the defect. In paper [12], the authors designed a bidirectional image difference algorithm to avoid the error detection of contour artifacts. In order to better locate the first row of captured image, a fast-computational algorithm based on image projection was given, which can convert 2D image searching into 1D feature matching. Paper [13] adopted laser scanning technology to detect overfill and underfill defects on upper surface of the deposited parts in the additive manufacturing process by comparing the existing point cloud with the presliced stereolithography (STL) model. In paper [14], Chervyakov et al. utilized two modified adaptive median filtering methods of impulse noise in images. The experiment showed potential application in processing satellite and medical imagery, geophysical data, and in other areas of digital image processing. Paper [15] provided a solution to the problem of distinguishing the defects and their own characteristics in robot 3D printing. The research findings can help to detect the defects online, improve the detection accuracy, and reduce the false detection rate without being affected by its own characteristics.
The abovementioned detection methods all applied image difference to extract defective patterns. Because the defective image and standard image collected in reality often have interferences with size, rotation, deformation, and other factors, whether the two images can be well registered will directly affect the accuracy of defect detection [16]. Image registration needs to be considered from the feature space, search space, interpolation method, search strategy, similarity measurement, and other aspects [6]. In addition, these inspection systems also need to meet some other requirements: (1) high-end and expensive line scan CCD camera; (2) special high-precision mechanical installation structure and lighting mode; (3) stable and reliable feeding platform; and (4) high-quality image data acquisition. These requirements limit the application range of the detection system [17–20]. For example, it is difficult to apply the online quality inspection system in the sheet-fed platform and the low-end printing production line because the vibration and interference of the platform are too large, and it is impossible to collect qualified images. For the research issue of low-quality printing image defect detection, the research results are few.
Currently, more and more researchers pay attention to the defect detection method based on machine learning [21]. Du used the deep learning method to improve the performance of X-ray image defect detection of automotive aluminum castings [22]. In paper [23], the defect detection of railway track fastener was studied by combining image processing and deep learning. However, labeling defect regions was time consuming, and it was difficult to collect enough defect samples for artificial neural network learning, which limited the application and promotion of deep learning in the field of defect detection. Paper [24] proposed an automatic inspection system with five-plane array charge-coupled device (CCD) cameras and four LED light sources in a closed environment. A support vector machine algorithm was adopted to classify defects based on the extracted features in candidate defect regions. In paper [25], Abul’khanov and Kazanskiy created visual and numerical tools to analyze a rough surface, through characterizing the rough surface by building its information pattern through imaging micro-roughnesses on the controlled surface and using the parameter value. Paper [26] designed a bridge cracks detection algorithm by using a modified active contour model and greedy search-based support vector machine. In paper [27], a novel cascaded autoencoder (CASAE) architecture was designed for segmenting and localizing defects. The defect regions of segmented results are classified into their specific classes via a compact convolutional neural network (CNN). Paper [28] adopted a single convolutional neural network (CNN) model that can extract effective features for defect classification without using additional feature extraction algorithms, and the proposed method can identify defect classes not seen during training by comparing the CNN features of the unseen classes with those of the trained classes. Paper [29] proposed a vision-based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. In addition, many other methods were also proposed for defect detection using machine learning, e.g., the generative adversarial networks [30] and reinforcement learning [31].
In summary, traditional defect detection methods based on image difference cannot be applied to low-quality images effectively and it is difficult to get enough defect samples to train a machine learning model for defect identification. In order to improve the adaptability of visual inspection system on a variety of different printing platforms (sheet-fed or roll-to-roll), in this paper, a new printed image database is established using a small CCD area array camera to collect images on a sheet-fed machine platform. Then, a new multi-edge feature fusion algorithm is proposed to adapt to defect detection in low-quality dataset.
3. Materials and Methods
The architecture of the proposed detection system consists of image registration, image sub-block, feature extraction, feature fusion, and feature matching, as shown in Figure 3. Initially, some basic image preprocessing methods are introduced and then we describe the detection algorithm using multi-edge feature fusion in detail. Because the variation of uneven illumination and deformation between samples can greatly affect the edge feature extraction, which will lead to false detection and missed detection, we adopt the feature fusion method to eliminate the influence of those interferences. Next, the detection evaluation criteria selected in this paper are described elaborately. Lastly, we present and analyze the experiment results.
3.1. Image Registration Using SIFT Feature
The obtained original image contains many interference factors, such as rotation and deformation [32]. The inconsistency of images needs to be corrected in advance. In this paper, SIFT feature is applied to eliminate these variations in original images, and Figure 4 shows the architecture of image registration using SIFT feature, including key points, matching images, and corrected images [33]. We cut off the redundant boundary with size of 50 pixels directly, and the image resolution is reduced from 800550 to 750500. The corrected images are as follows:where C is the corrected image, i and j are the coordinates of each pixel, R stands for the registered image, and k is the image sequence number. Then, we divide the whole image into several sub-blocks with resolution of 5050, as shown in image sub-block step in Figure 3.
3.2. Image Feature Extraction and Feature Matching
3.2.1. Feature Extraction Using Canny Edge Detection Operator
The mainstream edge detection operators include Roberts, Sobel, Prewitt, and so on [34]. Robert operator is sensitive to noise. Prewitt and Sobel operators have better detection performance on the image with gradual gray level and low noise, but for the image with mixed multi-complex noise, the processing effect is not ideal. The detection effect of the Canny operator is better than that of the gradient operator, which can detect the thin edge of the image. There are four processing steps consisting of noise reduction, gradient calculation, non-maximum suppression, and double threshold filtering. We can change the edge with multiple pixel width into a single pixel wide edge and remove the weak edge to retain the strong edge. Therefore, it is necessary to select the appropriate operator to detect the edge feature according to different environmental conditions and requirements. The output edge images are as follows:where B is the detected edge feature binary image, represent gradient amplitude images after non-maximum suppression, i and j are the coordinates of each pixel, and TL and TH represent high and low thresholds. Figure 5 presents the edge detection effect of the Canny operator, and the defective part is marked with a red circle.
(a)
(b)
(c)
(d)
3.2.2. Feature Similarity Matching Using Euclidean Distance
After obtaining the edge features, similarity with feature matching is used to judge the defect. Firstly, the edge feature image is reduced to one-dimensional feature vector. The formula is as follows:where F is the one-dimensional eigenvector of edge block feature, B represents every pixel on the binary edge feature image, m and n represent the row number and column number of feature image, and k is the image sequence number.
Then, Euclidean distance is used to match the similarity between two eigenvectors [17]. The formula of Euclidean distance iswhere is the Euclidean distance between two eigenvectors with , i represents every element in two eigenvectors, m and n represent the row number and column number of feature image, and Template and Test represent template and test eigenvectors, respectively.
According to the definition of Euclidean distance, smaller distance between two eigenvectors means greater similarity. We define the activation function of defect judgment as follows.
When the matching result is greater than the threshold value, the test eigenvector is recognized as defect and 1 is output.
3.2.3. Fluctuation Analysis of Feature Matching Similarity
Firstly, we defined feature similarity matching as four types: (a) single qualified pattern and multiple qualified patterns matching; (b) single qualified pattern and multiple defect patterns matching; (c) single defect pattern and multiple qualified patterns matching; and (d) single defect pattern and multiple defect patterns matching.
Figure 6 shows the corresponding matching results. According to analysis of matching results, the four matching results show uniform fluctuation and there is no obvious similarity difference between them. It is due to reflection of pattern surface, uneven external light, and mechanical vibration. These images with inconsistent fluctuation can lead to false detections. In the next section, we will utilize feature fusion method to eliminate matching similarity fluctuation between qualified patterns.
(a)
(b)
(c)
(d)
3.2.4. Template Establishment and Defect Segmentation
In order to eliminate the interferences as much as possible, we propose a multi-template edge feature fusion algorithm to increase the accuracy of defect identification. The architecture of the proposed method model is shown in Figure 7, and the formula of feature fusion is expressed as follows:where represents the template feature vector, stands for the defect image feature vector, and n is the sequence number of template images. Theoretically, the more the features are fused, the better the robustness of the template will be, and subsequent experiments will test the proposed template detection performance with different fusion sizes.
For visualization of defects, we extract the defect part using feature image difference, as shown in Figure 7. The defect feature is obtained using image difference between test feature and fusion feature. According to the block position number, we mark it in the original image to complete semantic segmentation and display the defect parts.
4. Results
It is difficult to collect enough defect samples in industrial production site. We ended up with 4035 complete qualified images and 135 defective images. Because most of the samples are defect free, we take 135 of them as a genuine class. Thus, the data used for test experiment contain 135 samples for each type and all the images are divided into 5050 sub-blocks. The dataset and some samples are shown in Table 1 and Figure 8, and the defective parts are marked with a red circle.
To demonstrate the effect of fusion size on the detection accuracy, some experiments under different fusion sizes were carried out, and the results are shown in Table 2. We designed seven different fusion sizes from 1 to 256. The two truth values represent genuine and defective patterns, respectively. Apparently, the detection accuracy increases significantly when the fusion size becomes larger. However, when the fusion size is larger than 160, the detection accuracy decreased. Therefore, the optimal fusion size is 160 for this case with accuracy of 95.18%, precision of 94.20%, and recall of 96.29%. When the feature fusion scale is too large or too small, it cannot achieve the ideal detection effect, such as using a single defect pattern for template feature generation.
In addition, we draw the similarity matching results with fusion size of 160 in Figure 9. The horizontal axis is the image number from 1 to 270, and the vertical axis is the Euclidean distance of feature matching from 0 to 1. From the distribution characteristics of red dots, there are only 5 missing detections with a matching result to 0 in the detection results of the first 135 defects on horizontal axis. On the contrary, 8 out of the last 135 qualified images are detected with matching distance not equal to 0.
Then, we select different thresholds to draw the P-R curve, as shown in Figure 10. The results are the same as shown in Table 2; the detection performance first increases to the green curve with fusion size 160 and then decreases. The front four curves are in a state of underfitting with insufficient amount of feature information learning and the last two curves are in a state of overfitting state with too much redundant information learning. Therefore, too much or too little feature fusion cannot achieve ideal recognition effect. In addition, we employ average precision (AP) and mean average precision (mAP) to evaluate the performance of the defect detection method. When fusion size is 160, the mAP and recall are 92.65% and 96.29%, respectively, as shown in Table 3.
Lastly, we test detection performance of the proposed method in extended dataset with 4035 patterns and 135 defects. Table 4 shows the experimental results in asymmetric dataset. The precision of small-sample dataset is only 31.47%, while it is up to 99.86% in large-sample dataset.
To demonstrate the effectiveness of the method proposed in this paper, a comparison is made for the other detection methods including non-fusion registration difference and convolutional neural network (CNN). The result is shown in Table 5. The method proposed in this paper achieves an accuracy of 93.09% which outperforms other methods. The reason for this result is that the low-quality images are not suitable for the traditional differential detection algorithm and it is different to provide sufficient defect training data for convolutional neural network, which leads to misclassification. In this paper, a new multi-edge feature fusion algorithm is used to recognize printing defects in low-quality datasets, which achieves a higher precision for the industrial printing image defect detection.
5. Conclusions
In this work, a new defect detection method using multi-edge feature fusion is proposed to improve the detection accuracy of low-quality printing images. The specific contributions are as follows:(1)We set up a new and more challenging print image dataset which consists of 4170 images and has more rotation, deformation, and uneven illumination changes, compared with the existing printing database.(2)The proposed multi-edge feature fusion algorithm can effectively distinguish pattern defects and interference changes.(3)Different feature fusion sizes will greatly affect the detection accuracy, and we also found that for all fusion scales, an optimal value exists for the detection accuracy; too large or too small amount of fusion information will reduce the overall detection performance of the system.
The current detection system mainly solves the problem of fine edge defect detection of low-quality printing image, while in further work, the detection system should be promoted to identify more types of defects. In addition, most samples of industrial printing products are qualified and authentic, which leads to unbalanced data types. In future work, how to collect enough defective images in industrial production field and how to use machine learning method to detect defective patterns need to be further studied.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.