Surface Defect Detection Method Based on Improved Semisupervised Multitask Generative Adversarial Network

Zhu, Lin; Baolin, Du; Xiaomeng, Zhao; Shaoliang, Fang; Zhen, Che; Junjie, Zhou; Shumin, Chen

doi:https://doi.org/10.1155/2022/4481495

Scientific Programming

On this page

Abstract Introduction Discussion Conclusions Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2022 | Article ID 4481495 | https://doi.org/10.1155/2022/4481495

Surface Defect Detection Method Based on Improved Semisupervised Multitask Generative Adversarial Network

Lin Zhu,¹Du Baolin,²Zhao Xiaomeng,¹Fang Shaoliang,¹Che Zhen,¹Zhou Junjie,¹and Chen Shumin¹

Academic Editor: Shah Nazir

Received01 Nov 2021

Revised08 Dec 2021

Accepted09 Dec 2021

Published19 Jan 2022

Abstract

The detection methods based on deep learning networks have attracted widespread interest in industrial manufacture. However, the existing methods are mainly trapped by a large amount of training data with excellent labels and also show difficulty for the simultaneous detection of multiple defects in practical detection. Therefore, in this article, a defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed for generating better image features and improving classification accuracy. First, the training data are manually labeled according to the types of defects, and the generative adversarial network (GAN) is constructed according to the reliable annotations about defects. Thus, a classification decision surface for the detection of multitype defects is formed in the discriminative network of GAN in an integrated manner. Moreover, the semisupervised samples generated by the discriminative network give the generative network feedback for enhancing the image features and avoiding gradient disappearance or overfitting. Finally, the experimental results show that the proposed method can generate high-quality image features compared with the classic GAN. Furthermore, this increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively.

1. Introduction

With the development of artificial intelligence and precision machining techniques, the quality requirements of industrial products are also increasing. However, machining defects are inevitable in industrial manufacture. Since machined surface defects may have a disadvantageous effect on the properties and quality of industrial products, the defect detection and analysis of the industrial products are regarded as an important process for controlling the quality and increasing the commercial value. Recently, machine vision technology has been used extensively to test the production quality, such as defect detection [1], industrial diagnostics [2], and other fields [3]. At the same time, to increase the accuracy of defect detection, academic researchers and the industry have developed many effective detection methods and conducted many research activities, such as deep neural network (DNN) [4, 5], logistic regression (LR) [6], deep belief network (DBN) [7], and other methods [8]. Among these approaches, with the advancement and research of deep learning techniques, the defect detection methods based on DNN have become popular, which automatically learn image features from a large scale of training data through the multilayer neural networks [9, 10]. Therefore, the methods have automatic feature extraction and powerful learning capabilities and have been successful in various fields of power electronics [11], microelectronics [6], and other fields [12]. The feature maps of different layers and deep CNN were used by Chen et al. [13] to classify the fastener defects of the support devices, and the public dataset with excellent labels was used in the experiments. Mei et al. [14] proposed a method to localize and detect defect-free data for model training, and the performance of model was evaluated on the KTH-TIPS2 datasets. He et al. [15] proposed an effective defect detection method based on DNN and focused on industrial manufacture: steel plate defect inspection. The performance of DNN was evaluated on the defect datasets: NEU-DET and NEU-CLS, and each class of the defect datasets has 300 images. The defect detection methods based on DNN have been extensively used in industrial application. However, the methods have their own disadvantages. The DNNs, which perform entirely on benchmark datasets, may perform poorly in practical detection [16, 17]. Generally, the advantageous performance of these detection methods based on DNN depends on a large amount of training data with excellent labels [18]. However, collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and the types of the samples are difficult to balance. Most prior works have been limited by small training sample sizes, which impact the power to detect effects and the reproducibility of results.

For the problem of small samples, many academic researchers use the image enhancement method to expand the training samples. Three image enhancement approaches based on the characteristics of the synthetic image were designed by Ding et al. [19] to expand the training dataset. These methods could increase the performance of CNN in defect detection. However, this method can only expand the single image, and the quality of the generated samples is poor, and the training dataset expansion effect is not ideal. As a solution, the generative adversarial network (GAN) can quickly reconstruct or generate images with various patterns or enhance the quality of the training dataset, which has displayed hopeful performance in generating sample images [20, 21]. At present, the GAN has been extensively used in different fields, such as image restoration [22] and speech enhancement [23]. A deep convolutional GAN (DCGAN) was proposed by Radford et al. [24], which normalizes the input in batches, and it has achieved better results in image generation. Although GAN has gained popularity, it still has some fatal deficiencies. The GAN has some problems that the network structure is unstable, and training GAN itself is difficult. To solve the problem, several methods [25, 26] were proposed. An efficient detection approach based on improved GAN (iGAN) was developed by Wang et al. [27] for machined surfaces. The iGAN is similar to an extension of the classic GAN. On this basis, many methods have produced satisfying efficacies in semisupervised image classification by improving the structure of the DCGAN model. An improved conditional DCGAN model was proposed by Yang et al. [28] which could effectively improve the accuracy of image recognition, but this method was considered to be commercially time-consuming. The improved GAN has a more stable architecture than the classic DCGAN by applying some constraints on GAN. Thus, it is necessary to optimize the constraints of the DCGAN. The information maximizing generative adversarial net (InfoGAN) was designed by Chen et al. [29] to learn entangled representations in an unsupervised method. An adversarial learning approach was developed by Hu et al. [30] to increase the accuracy of semantic segmentation by connecting the cross-entropy loss with the adversarial loss of the proposed method. For the problem that the existing methods mostly depend on weakly labeled images, Zhang et al. [20] proposed a semisupervised GAN (SSGAN) with two subnetworks to expand the training dataset, and the proposed method could improve semantic segmentation accuracy with fewer annotations. The performance of the above methods is enhanced to a certain extent compared with CNN. The GAN has been widely used in many related detection tasks and requires significant computational resources in software implementations and a large amount of data for training. Because training GAN itself (a deep generative model) will rely on sufficient samples, if it is not ensured, GAN will be overfitted and will collapse. In other words, these techniques pursue prediction accuracy at the expense of computational efficiency. Moreover, the discriminator of the GAN is mainly dichotomous classification, which is challenging to apply on the detection of multitype defects. However, the application of the multitask GANs, exceptionally for surface detection using small samples, is still rare, which deserves more attention.

In practical detection, the existing methods are mainly trapped by a large amount of training data with excellent labels and also show difficulty for the simultaneous detection of multiple defects. If the number of the training datasets is not sufficiently large and the training datasets are imbalanced, training the defect detection method from scratch is likely prone to overfitting in the detection of multitype defects. Therefore, in this article, a surface defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed to reduce overfitting caused by the lack of training samples. The primary contributions of this paper are summarized as follows:(1)The training dataset is manually labeled according to the type of the defect, and a classification decision surface for the detection of multitype defects is formed in the adversity-discriminator link of iSSMT-GAN in an integrated manner. The discriminative network inputs both real and fake samples and distinguishes whether its input is real or not. Its output result is used as the basis for the classification decision surface for the detection of multitype defects.(2)The structure of iSSMT-GAN is improved, and the multitask noise is introduced in the generative network of GAN. Moreover, the semisupervised samples generated by the discriminative network give the generative network feedback to guide the generative network, which is an efficient way to avoid overfitting. More specifically, in the discriminator of iSSMT-GAN, the probability of judging the authenticity of the multitask sample output by the discriminator is returned to the generator, and the resulting latent code from the discriminator can be fed back to guide the generative model.

This article is organized in the following manner. Section 2 indicates the problem formulation and motivation and introduces the network structure of GAN. Section 3 presents the improved semisupervised multitask generative adversarial network (iSSMT-GAN), and the semisupervised samples generated by the discriminative network give the generative network feedback. The experiment results are reported to verify the efficacy of the proposed methodology in Section 4. Finally, Section 5 gives the conclusions and future work.

2. Problem Formulation and Motivation

2.1. Problem Formulation

In practical manufacture, the industrial products usually include multitype defects, such as bump defects, hole defects, burr defects, oxidation defects, and so on. Collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and it leads to a series of difficulties in the defect detection process due to imbalanced datasets. The main problems of surface defect detection are summarized as follows:(1)In practical detection, the detection methods based on DNN are prone to overfitting and weak generalization ability during training due to the unbalanced training dataset and the small number of samples. It further leads to low accuracy and recall in actual product testing [6, 10, 11].(2)At this stage, the discriminator of the GAN is mainly dichotomous classification, which is challenging to apply on the detection of multitype defects. However, the detection of multitype defects is a significant topic in actual industrial applications.(3)Although many experts and scholars have adopted GAN for image defect detection at this stage, the problem of gradient disappearance or overfitting of the GAN is straightforward to occur during the learning process of data.

In this article, a defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed for generating better image features and improving classification accuracy. First, the iSSMT-GAN is used to predict the potential distribution of the training dataset and reconstruct the image of multitype defects. By integrating the advantages of GAN, it can solve the problems, such as small sample size, not noticeable features, and weak generalization ability in practical detection. From the perspective of sample feature quality and multitask classification recognition, the classification accuracy and recall are improved.

2.2. Generative Adversarial Network

The GAN is a framework for estimating generative models through an adversarial process [31]. The GAN framework contains two separate networks, the generative network G and the discriminative network D. The basic framework of GAN is shown in Figure 1.

In Figure 1, the generative network G is trained to generate the fake samples from the latent variable . In GAN, the generating network can quickly learn the target data distribution. The objective function of the generative network is shown in the following equation [31]:where refers to the calculation of the expected values, is the output of the discriminative network, and and refer to the distribution of real and generated sample data, respectively.

The discriminative network inputs both real data and generated data and distinguishes whether its input is real or not. The objective function of the discriminative network is shown in the following formula:

Therefore, the optimization problem of GAN is a minimization-maximization problem. The discriminative network and the generative network play the two-player mini-max game with the value function [19]:where represents the actual sample data used for training, is the distribution of samples, is the output result of the discriminative network, is the noise of the input, is the distribution of known noise , and represents the probability that the generated data are differentiated to be an actual sample after passing through the discriminator.

3. Improved Semisupervised Multitask Generative Adversarial Network

For generating better image features and improving the classification accuracy, an improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed to reduce overfitting caused by the lack of training samples. In the iSSMT-GAN, the multitask noise is introduced in the generator of GAN according to the reliable annotations about multitype defects. In generating multitask samples, a low-rank space is formed, and a classification decision surface is formed in the adversarial discriminator of GAN. At the same time, the resulting latent code from the discriminative network can be fed back to guide the generative model. The framework of the improved semisupervised multitask generative adversarial network (iSSMT-GAN) is illustrated in Figure 2.

The framework of iSSMT-GAN contains four parts: the generative network, the discriminative network, the semisupervised samples generated by the discriminative network giving the generative network feedback, and the classification decision surface for the detection of multitype defects.(1)The generative network G is constrained with MSE, and the prediction of total variation loss and image features extracted by CNN, which forces the generated data to be more realistic [24] and the batch normalization layer can be used to solve the problems of gradient disappearance and gradient explosion.(2)The discriminative network D inputs both real and fake data and distinguishes whether its input is real or not, and the output result is used as the basis for the classification decision surface for the detection of multitype defects. At the same time, the semisupervised samples generated by the discriminative network give the generative network feedback.(3)The classification decision surface for the detection of multitype defects performs multitask classification tasks based on the scene-prejudged defect labels and the output results of the discriminative network. Thus, a classification decision surface for the detection of multitype defects is formed in the adversity-discriminator link of iSSMT-GAN in an integrated manner.

The difference between the proposed iSSMT-GAN and the classic GAN is summarized as follows:(1)In the discriminative network of iSSMT-GAN, a classification decision surface for the detection of multitype defects is formed by labeling sample defect images and supervising the discriminator.(2)According to the annotation result and the results of the discriminative network, the semisupervised samples generated by the discriminative network give the generative network feedback to guide the generative network, which is an efficient way to avoid overfitting, so that many more characteristic sample images are generated through the generation network.

3.1. Multitask Discriminative Network

In the iSSMT-GAN, a multitask discriminative network is built by constructing a classification decision surface for the detection of multitype defects according to reliable annotations about multitype defects. Firstly, the discriminator makes a preliminary judgment based on the manual labels of defects. Then, the GAN performs dichotomous classification tasks in different defects. Finally, a classification decision surface for detecting multitype defects is formed in an integrated manner. The schematic diagram for the multitask discriminative network is shown in Figure 3.

At the same time, the multitasking noise is treated as an input variable to the generative network. In the training process, the GAN can generate samples closer to the real samples distribution from the given multitasking noise . The multitasking noise is introduced in the detection task facing multiple scene modes, and a low-rank space shared by multiple tasks is formed in generating multitask samples.

The discriminative network is constructed using a convolution layer and fully connected layers, and LeakyReLU is selected as each convolution layer’s activation function. Each convolutional block consists of three major components, including a convolutional layer, a LeakyReLU activation layer, and a batch normalization layer [32].

The loss function is an essential part of the GAN, and it can directly determine the performance of the network model. According to the structure of GAN, the total loss value can be calculated as the sum of the counter loss and content loss. Its application is shown in the following formula:where is the counter loss, is the content loss, and is a constant.

To optimize the parameters of our proposed model, the loss function using mean square error (MSE) can be defined as follows [23]:where is the real image, is the generated image, and is the number of training datasets.

The optimization of the model is achieved by minimizing the confrontation loss and the residual feedback of MSE. The loss function of GAN is shown as follows:

3.2. The Semisupervised Sample Feedback

The structure of iSSMT-GAN uses a simple factored continuous input noise vector . Therefore, the noise vector may be used by the generative network in an extremely involved mode, causing various scales of the noise vector to be discordant to the semantic characteristics of the image [24].

In this article, the input noise vectors are divided into two parts instead of using a single incompressible noise vector: (i) represents the source of unstructured noise; (ii) represents the latent code. Define the set of methodic latent variables by . Then, a factored distribution can be assumed, given by the following formula [24]:

For the concise and clear notation, the latent code will be used to represent the connection of all latent variables .

An unsupervised way is proposed to discover these latent factors. Then, the input to the generative network is the latent code and the incompressible noise, and the form of generative network becomes . However, the generative network can neglect the attached latent code by searching a solution that meets the following condition:

Moreover, in order to generate or reconstruct the images that are more suitable for defect detection, the semisupervised samples generated by the discriminative network are fed back to the generative network. Therefore, the defect detection is performed after high-resolution images are generated, and the accuracy of detection is improved by improving the structure of GAN model. The structure of the semisupervised sample feedback is shown in Figure 4.

In iSSMT-GAN, the structure of the semisupervised sample feedback is formed according to the discriminant results. In the discriminator of iSSMT-GAN, the probability of judging the authenticity of the multitask sample output by the discriminator is returned to the generator and the semisupervised samples generated by the discriminative network give the generative network feedback. In the feedback process, the classification result after the image label is carried corresponds to the signal input to the generator after scene prediction, which has a semisupervised effect.

4. Experiments and Discussion

In this section, the experiments are divided into two parts: Section 4.1. presents the experimental methods and Section 4.2. presents the experimental results.

4.1. Experimental Methods

4.1.1. Dataset

To verify the efficacy of the proposed method, comparative evaluation experiments using the NEU-DET dataset and the PCB dataset (self-built dataset) were conducted, and the performance of the proposed iSSMT-GAN is further tested on the NEU-DET dataset and PCB dataset. NEU surface defect is a public defect classification dataset [16], including six different classes from hot-rolled steel plates, such as inclusion, patches, crazing, rolled-in scales, scratches, and pitted surface. Each class has 300 images. Examples of defect images are shown in Figure 5.

(a)

(b)

(c)

(d)

(e)

(f)

However, collecting defect images at large scale is expensive due to the particularly low probability of defect occurrence, and the types of the samples are difficult to balance. The highly imbalanced datasets are common in many pattern recognition tasks. The severe class imbalance of training datasets is still a significant point that needs to be addressed. The PCB dataset is collected by AOI detection equipment from real detection scenarios of a company. The PCB dataset (self-built dataset) contains 3569 images with a size of 161 × 161. There are six types of defects, including open circuit defects, bump defects, hole defects, residual copper defects, burr defects, and oxidation defects. We manually label 3569 images at first. The dataset contains 2,923 training images and 646 test images of 6 classes. Table 1 shows that this PCB dataset is small in size and the dataset is highly unbalanced. The size of all the images was adjusted to 96 × 96. Figure 6 shows the different types of defects for the dataset.

(a)

(b)

(c)

(d)

(e)

(f)

4.1.2. Experimental Environment

The data are employed to verify the proposed method. The experimental environment is described as follows: deep learning open source framework PyTorch [6]. The experiments are performed on the computer with Intel Core i7 processor at 3.4 GHz with 8GB memory.

4.1.3. Evaluation Indicators

In this experiment, model discrimination was assessed using machine learning evaluation metrics, including precision, F₁ score, accuracy, and recall. Among them, accuracy is used to evaluate the method's ability to accurately find samples. The recall is used to evaluate the method's ability to find a certain type of sample in the dataset. These indexes are defined as follows:where TN, FN, TP, and FP represent the number of true negatives, false negatives, true positives, and false positives, respectively.

4.2. Experiment Result

Comparative evaluation experiments were conducted using PCB dataset. Table 1 shows that this PCB dataset is small in size and the dataset is highly unbalanced. In response to the problems, the proposed iSSMT-GAN and the traditional DCGAN were used to expand the size of the training dataset and generate a balanced training dataset. Finally, the defect detection approaches based on deep learning were applied for the expanded datasets in the experiments.

4.2.1. Training of the Proposed iSSMT-GAN

To verify the efficacy of the proposed iSSMT-GAN to generate images, the classic DCGAN [24] and the proposed iSSMT-GAN are trained on each category of training data for 2000 epochs, respectively. The number of image channels was set to 3. In the training process, the number of training epochs was 2000, the optimizer used in this experiment was Adam, the initial learning rate was 0.002, and a batch size of 25 was used.

The iSSMT-GAN and the DCGAN were used for enhancing dataset from the real sample images of patches of the NEU-DET dataset, respectively. Figure 7 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 1000 epochs. Figure 8 shows the generated images using iSSMT-GAN and DCGAN on the NEU-DET dataset after training at 2000 epochs.

(a)

(b)

(a)

(b)

The iSSMT-GAN and the DCGAN were used for enhancing the dataset from the real sample images of hole defects of PCB dataset, respectively. Figure 9 shows the real sample images of hole defects in the dataset. Figure 10 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 1000 epochs. Figure 11 shows the generated images using iSSMT-GAN and DCGAN on the PCB dataset after training at 2000 epochs.

(a)

(b)

(a)

(b)

Figures 10 and 11 show that since the generative network and the discriminative network are continually changing during training, the generated image will change. The quality of samples suggests that the quality of training data impacts the generated images performance of the iSSMT-GAN and DCGAN methods. Figures 10(a), 10(b), 11(a), and 11(b) show that the image features generated by proposed iSSMT-GAN are more apparent and more obvious than the images generated by DCGAN using the NEU-DET dataset and the PCB dataset, respectively. From Figures 10(a) and 11(a), when the number of training epochs is 1000, there will be a gradient explosion phenomenon. Increasing the number of images in training data can eliminate overfitting, strengthen generalizability, and improve the performance of the training dataset.

In contrast, the generated images using iSSMT-GAN disclose much-detailed high-resolution features, and the image quality is uniform with the training data. It has been proven that selecting the appropriate training images in the iSSMT-GAN can efficiently improve the dataset quality. The iSSMT-GAN approach requires a long training time, and real-time capability was influenced by the computational resources. Figure 12 shows the loss results of the generator and the discriminator on the NEU-DET dataset. Figure 13 shows the loss results of the generator and the discriminator on the PCB dataset.

(a)

(b)

(a)

(b)

The proposed method is also compared with that based on DCGAN [24] in the experiments. Since 2018, the DCGAN has been successful in addressing the problem of limited labeled training samples in the CIFAR-10 dataset and ImageNet-1k dataset. From Figures 12(a) and 13(a), the experiment results indicate that the DCGAN model presents a phenomenon of overfitting during training. Because training DCGAN itself (a deep generative model) will rely on sufficient samples, if it is not ensured, GAN will be overfitted and will collapse. From Figures 12(a) and 13(a), the experiment results show that the training DCGAN model suffers from a certain degree of overfitting, and the experiment results are not unexpected. On the one hand, this may be due to the small size of the training dataset. On the other hand, the training dataset contains more complex data than the CIFAR-10 dataset. In the comparative experiment, the proposed iSSMT-GAN is trained using the same dataset. The proposed method displays better results under the same conditions.

Figures 12 and 13 show the loss changes of the generator and discriminator during training. From Figures 12(a) and 13(a), the training loss values of the generator and discriminator oscillate more and more violently. Because the discriminator and the generator are in a dynamic game, the training process of iSSMT-GAN is commonly unsteady. From Figures 12(b) and 13(b), it is clear that the generator and the discriminator improved via the adversarial process. The training loss values of the generator and discriminator converge well, despite the fluctuating phenomenon in the process of training. At the same time, the generator learns to reconstruct or generate more realistic images, which become pyramidally difficult for the discriminative network to discriminate as not real, as observed from the downward trend in the loss values of the discriminative network and upward trend in the loss values of the generative network curves in early epochs. In the iSSMT-GAN, the loss value decreased significantly during the training period of 0–500 epochs and then decreased steadily. The loss value of the discriminator closer to 0 indicates that the discriminator can discriminate the real image and the generated image.

In practice, we believe that the improved discriminative network can improve the performance of the generative network to some extent.

4.2.2. Defect Detection Experiment

In the experiment, the PCB dataset was separated into two parts, including the training dataset and the test dataset. At the same time, the proposed iSSMT-GAN and DCGAN are used to increase the size of the dataset. The distribution of sample sizes in the dataset is shown in Table 1. In contrast, the generated image method can change the distribution of the dataset.

To verify the efficacy of two different training datasets expanded by the proposed iSSMT-GAN and the classic DCGAN [24], respectively, the defect detection models, such as AlexNet-based transfer learning [9], the VGG-19 model [10], the MobileNet v3 model [12], and the RegNet model, were trained with the two different datasets. From Table 1, the expanded dataset has the problem of the imbalance problem in the training data. To avoid overfitting, we also tested the detection model on the imbalanced datasets by 10-fold cross-validation. The number of training epochs was set as 300, the batch size was 64, and the learning rate was 0.002.

The training accuracy and loss value of AlexNet-based transfer learning training are shown in Figure 14. The training accuracy and loss value of VGG-19 model training are shown in Figure 15. The testing performance of each model is shown in Table 2.

(a)

(b)

(a)

(b)

From Figures 14 and 15 and Table 2, the accuracy increases with epoch number and eventually reaches the maximum (98.43%, 93.10%, 95.31%, and 93.75%) in the training process of AlexNet-based transfer learning, the VGG-19 model, the MobileNet v3 model, and the RegNet model, respectively. From Table 2, the experiment results show that the classification accuracy of AlexNet-based transfer learning method is 98.43%, in the training dataset expanded by iSSMT-GAN. The classification accuracy of AlexNet-based transfer learning method is 95.31%, in the training dataset expanded by DCGAN. Figures 14(a) and 14(b) show that the training loss values of the AlexNet-based transfer learning training converge well, and the loss value at the beginning of the model training is tremendous, but as the training process progresses, the loss value extends to decrease on the whole. On the training dataset, the VGG-19 model begins to converge after about 75 epochs, and the convergence process is stable.

This increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively. In the dataset expanded by iSSMT-GAN, AlexNet-based transfer learning showed the F₁ score of 0.9817, reflecting a superior recall performance and precision. Therefore, the dataset expanded by the proposed iSSMT-GAN turns out to be the best option, considering the performance and accuracy of defect detection.

In summary, through the analysis of classification quality and classification cost indicators, the proposed system can not only generate better image feature but also improve the classification accuracy. The proposed iSSMT-GAN model displays better results under the same conditions. In contrast, the value of classification accuracy in VGG-19 model only increases slightly from 90.62% to 93.10%. In general, the developed iSSMT-GAN model is efficient for generating better image features compared with DCGAN and improving the classification accuracy using the same dataset.

5. Conclusions

In this article, a defect detection method based on an improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed. The training data are classified and manually labeled according to the type of defects, and the iSSMT-GAN is constructed according to reliable annotations about defects. Thus, a classification decision surface for the detection of multitype defects is formed in the discriminative network of GAN. The applicability and feasibility of the proposed iSSMT-GAN model are verified on the PCB dataset (self-built dataset) to expand the size of the training dataset, and this increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively. Experiment results are presented to verify the proposed method. This research mainly focuses on the defect detection on the PCB surface. It is noted that micron-level defects are widespread in industrial products. Our future research will focus on reconstructing or generating micron-level image, so that the test approach adapts to the classification of micron-level defects.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

The authors are grateful for the financial support from the Guangdong Research Institute of China Engineering Science and Technology Development Strategy of China (grant no. 2021-GD-11), Basic Research Project of Guangzhou, China (grant no. 202002030300), and Guangdong Provincial Fund Project, China (grant nos. 2019A1515011376 and 2021A1515011615).

References

J. Yang, S. Li, Z. Wang, H. Dong, J. Wang, and S. Tang, “Using deep learning to detect defects in manufacturing: a comprehensive survey and current challenges,” Materials, vol. 13, no. 24, 2020.
View at: Publisher Site | Google Scholar
R. O. Dogan, H. Doan, C. Bayrak, and T Kayikcioglu, “A two-phase approach using mask R-CNN and 3D U-Net for high-accuracy automatic segmentation of pancreas in CT imaging,” Computer Methods and Programs in Biomedicine, vol. 207, no. 1, pp. 1–8, 2021.
View at: Publisher Site | Google Scholar
S. Deitsch, V. Christlein, S. Berger et al., “Automatic classification of defective photovoltaic module cells in electroluminescence images,” Solar Energy, vol. 185, no. 1, pp. 455–468, 2019.
View at: Publisher Site | Google Scholar
Z. Qiu, F. Wang, and Z. Pan, “Adaptive adjustment object detection algorithm under multiple mechanisms based on GAN,” Scientific Programming, vol. 2021, Article ID 5875320, 7 pages, 2021.
View at: Publisher Site | Google Scholar
J. Park, D. K. Han, and H. Ko, “Fusion of heterogeneous adversarial networks for single image dehazing,” IEEE Transactions on Image Processing, vol. 99, no. 1, pp. 4721–4732, 2020.
View at: Publisher Site | Google Scholar
R. Ren, T. Hung, and K. C. Tan, “A generic deep-learning-based approach for automated surface inspection,” IEEE Transactions on Cybernetics, vol. 99, no. 1, pp. 1–12, 2017.
View at: Publisher Site | Google Scholar
W. Liu, Z. Liu, H. Wang, and Z. Han, “An automated defect detection approach for catenary rod-insulator textured surfaces using unsupervised learning,” IEEE Transactions on Instrumentation and Measurement, vol. 99, no. 1, pp. 8411–8423, 2020.
View at: Publisher Site | Google Scholar
U. Fiore, A. D. Santis, F. Perla, P. Zanetti, and F. Palmieri, “Using generative adversarial networks for improving classification effectiveness in credit card fraud detection,” Information Sciences, vol. 479, no. 1, pp. 448–455, 2019.
View at: Publisher Site | Google Scholar
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proceedings of the International Conferennce on Neural Information Processing Systems, pp. 1097–1105, Curran Associates Inc., Lake Tahoe, NV, USA, December 2012.
View at: Publisher Site | Google Scholar
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” Computer Science, vol. 1, no. 3, pp. 1–10, 2014.
View at: Google Scholar
E. F. S. Filho, R. M. Prates, A. P. Marotta, R Cruz, R. P. Ramos, and J. S. Cardoso, “Insulator visual non-conformity detection in overhead power distribution lines using deep learning,” Computers & Electrical Engineering, vol. 78, no. 1, pp. 343–355, 2019.
View at: Publisher Site | Google Scholar
A. Howard, M. Sandler, B. Chen et al., “Searching for MobileNetV3,” in Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Seoul, Korea (South), November. 2019.
View at: Publisher Site | Google Scholar
J. Chen, Z. Liu, H. Wang, A. Nunez, and Z. Han, “Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network,” IEEE Transactions on Instrumentation and Measurement, vol. 67, no. 2, pp. 257–269, 2018.
View at: Publisher Site | Google Scholar
S. Mei, H. Yang, and Z. Yin, “An unsupervised-learning-based approach for automated defect inspection on textured surfaces,” IEEE Transactions on Instrumentation and Measurement, vol. 67, no. 1, pp. 1266–1277, 2018.
View at: Publisher Site | Google Scholar
Y. He, K. Song, Q. Meng, and Y. Yan, “An end-to-end steel surface defect detection approach via fusing multiple hierarchical features,” IEEE Transactions on Instrumentation and Measurement, vol. 69, no. 4, 2019.
View at: Publisher Site | Google Scholar
Y. Dogan and H. Y. Keles, “Semi-supervised image attribute editing using generative adversarial networks,” Neurocomputing, vol. 401, no. 1, pp. 338–352, 2020.
View at: Publisher Site | Google Scholar
K. Kyeong and H. Kim, “Classification of mixed-type defect patterns in wafer bin maps using convolutional neural networks,” IEEE Transactions on Semiconductor Manufacturing, vol. 31, no. 3, pp. 395–402, 2018.
View at: Google Scholar
C. Hu and Y. Wang, “An efficient convolutional neural network model based on object-level attention mechanism for casting defect detection on radiography images,” IEEE Transactions on Industrial Electronics, vol. 67, no. 12, 2020.
View at: Publisher Site | Google Scholar
J. Ding, B. Chen, H. Liu, and M. Huang, “Convolutional neural network with data augmentation for SAR target recognition,” IEEE Geoscience and Remote Sensing Letters, vol. 13, no. 3, pp. 364–368, 2016.
View at: Publisher Site | Google Scholar
G. Zhang, Y. Pan, and L. Zhang, “Semi-supervised learning with GAN for automatic defect detection from images,” Automation in Construction, vol. 128, no. 1, 2021.
View at: Publisher Site | Google Scholar
Z. Zhai, “Auto-encoder generative adversarial networks,” Journal of Intelligent and Fuzzy Systems, vol. 35, no. 3, pp. 1–7, 2018.
View at: Publisher Site | Google Scholar
B. W. Tseng and P. Y. Wu, “Compressive privacy generative adversarial network,” IEEE Transactions on Information Forensics and Security, vol. 15, 2020.
View at: Publisher Site | Google Scholar
H. Li, Y. Xu, D. Ke, and K. Su, “μ-law SGAN for generating spectra with more details in speech enhancement,” Neural Networks, vol. 136, no. 1, pp. 17–27, 2021.
View at: Publisher Site | Google Scholar
A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv e-prints, vol. 1511, Article ID 06434, 2018.
View at: Google Scholar
S. M. Iranmanesh and N. M. Nasrabadi, “HGAN: Hybrid generative adversarial network,” Journal of Intelligent and Fuzzy Systems, vol. 1, no. 1, pp. 1–12, 2021.
View at: Publisher Site | Google Scholar
D. D. Pierre-Luc and K. Nathan, “Quantum generative adversarial networks,” Physical Review A, vol. 98, no. 1, Article ID 012324, 2018.
View at: Publisher Site | Google Scholar
Q. Wang, R. Yang, C. Wu, and Y. Liu, “An effective defect detection method based on improved Generative Adversarial Networks (iGAN) for machined surfaces,” Journal of Manufacturing Processes, vol. 65, no. 5, pp. 373–381, 2021.
View at: Publisher Site | Google Scholar
M. Yang, “Underwater image enhancement based on conditional generative adversarial network,” Signal Processing: Image Communication, vol. 81, no. 1, pp. 1–10, 2019.
View at: Publisher Site | Google Scholar
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets,” in Proceedings of the Neural Information Processing Systems (NIPS), Spain, December 2016.
View at: Google Scholar
H. Hu, X. Zhai, G. Fei, and G. Hu, “ITDPM: an internet topology dynamic propagation model based on generative adversarial learning,” Scientific Programming, vol. 2021, Article ID 2390466, 9 pages, 2021.
View at: Publisher Site | Google Scholar
I. J. Goodfellow, “Generative adversarial networks,” Advances in Neural Information Processing Systems, vol. 3, no. 1, pp. 2672–2680, 2014.
View at: Publisher Site | Google Scholar
M. Wang and W. Deng, “Deep face recognition with clustering based domain adaptation,” Neurocomputing, vol. 393, no. 1, pp. 1–14, 2020.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2022 Lin Zhu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

1014

Downloads

781

Citations