Abstract
In recent years, learning based machine intelligence has aroused a lot of attention across science and engineering. Particularly in the field of automatic industry inspection, the machine learning based vision inspection plays a more and more important role in defect identification and feature extraction. Through learning from image samples, many features of industry objects, such as shapes, positions, and orientations angles, can be obtained and then can be well utilized to determine whether there is defect or not. However, the robustness and the quickness are not easily achieved in such inspection way. In this work, for solar panel vision inspection, we present an extreme learning machine (ELM) and moving least square regression based approach to identify solder joint defect and detect the panel position. Firstly, histogram peaks distribution (HPD) and fractional calculus are applied for image preprocessing. Then an ELM-based defective solder joints identification is discussed in detail. Finally, moving least square regression (MLSR) algorithm is introduced for solar panel position determination. Experimental results and comparisons show that the proposed ELM and MLSR based inspection method is efficient not only in detection accuracy but also in processing speed.
1. Introduction
Over the past decade, machine vision has been widely applied in industrial inspection, especially in electric and electronic industrial inspection [1–3]. Superior to manual inspection, the benefit of machine vision detection is that no direct contact is required between the inspector and the target. Therefore, automatic nondestructive inspection can be achieved in machine vision detection system.
Since the accuracy of traditional vision inspection depends much on the effect and the stability of image processing technique, the traditional inspection way will lose its’ functionality when facing the changing environment or the altered imaging condition. Therefore, not only processing the images of industry objects but also learning the features (shapes, positions, and orientations) from image samples is required. Actually, with the fast development of computer vision, recently, there have appeared some works [4, 5] for machine learning based vision inspection.
In modern photovoltaic electronic industry, for the solder joint of solar PCB is so fine, the vision based solder joint defect identification is challenging and thus needs to be investigated in depth [6, 7]. In addition, during the manufacture of solar panel, there is a strong demand for automatic position determination of the solder panel [8, 9]. In this work, in the face of the two major challenges, we present a set of defect identification and position determination methods.
So far, Capson and Eng [10] have used a tiered-color illumination approach for solder joint’s inspection. Matsuyama et al. [11] also design a system for automated solder joint inspection. In addition, Chiu and Perng [12] utilize the reflection area of solder joint to describe the image features, and Tae-Hyeon et al. [13] develop a vision inspection system for the classification of solder joint. Moreover, unlike previous methods, Zhang and Liu in [14] propose utilizing the principle of ELM [15] with least square fitting to achieve vision detection on solder joint and solar panel. In this work, we extend and improve the previous work [14] in three aspects: histogram peaks distribution (HPD) [16] and fractional calculus [17–20] are adopted to improve the effect of image preprocessing and initial edge extraction; instead of least square fitting, MLSR is adopted to enhance the accuracy of edge estimation of the solar panels; more vision inspection experiments and comparisons of more solar PCB images are implemented.
Specifically, for solder joint defect identification, we take histogram peaks distribution (HPD) [16] with morphological operation to extract image features and then apply extreme learning machine (ELM) [21–24] to classify the input images. As for solar panel position determination, accurate edges location is prerequisite. To achieve this, we extract the initial edge points by a novel signal processing method, fractional calculus [16, 22–24], and then utilize MLSR algorithm [25, 26] to refine initial edges. Once the final edges of solder panel are acquired, the position parameters, position shift, deflection angle, and edge lengths, are easily determined. The overall proposed scheme of solder joint defect detection and solar panel position determination is illustrated in Figure 1.
The paper is organized as follows. Section 2 discusses the related techniques of image preprocessing and feature extraction used for defect identification and position detection. Section 3 describes how to use ELM to recognize solder joint defect in detail. Section 4 presents utilizing MLSR algorithm to achieve solder panel position determination. The abundant experimental results and comparisons are given in Section 5. Section 6 makes a short conclusion and future research prospective.
2. Image Preprocessing and Feature Extraction
In this section, we firstly make a detailed discussion on using HPD and morphological operation to preprocess the images of solder joint. And then we introduce fractional differential to extract initial edges of solar panel. Finally, we also elaborate on the specific feature extraction method used for solder joint defect classification.
2.1. Image Preprocessing Based on HPD and Morphological Operation
In machine vision inspection system, image preprocessing, including noise removing, image binarization, and edge detection, is the basic guarantee for following detection. However, due to illumination or view angle changing and noise intervention, such image preprocessing task under complex environment is always challenging. Fortunately, in solar panel vision detection system, the illumination and the view angle are under control. And imaging noise can be suppressed in advance by camera calibration beforehand. Thus the preprocessing effect of solar panel images can be guaranteed.
Image thresholding is an effective technique for image binarization. Due to the unevenness of the lighting distribution, threshold finding is not simple and straightforward. How to acquire an appropriate threshold becomes the key issue for image binarization. Here, we take HPD [16] to find an accurate threshold for solder panel image binarization. HPD threshold search can handle several histogram distributions, which are illustrated in Figure 2.
(a)
(b)
(c)
(d)
(e)
(f)
Morphological operation can obtain the backbone of the image object which can unify a host of techniques such as noise suppression, shape analysis, skeletonizing, and other related topics [27]. After thresholding, the basic morphology operations, dilation and erosion, are usually used to process binary images. In addition, the other two important morphological operations, closing and opening, are also typically used after thresholding. The closing operation smooths the contours of the image to eliminate narrow and long gaps or remove holes or fill contours. The open operation is used to eliminate tiny image objects or separate objects connected to narrow areas.
For solder joint images, we firstly convert image color space from RGB to Ycbcr. Then in -intensity subspace, we utilize HPD algorithm to obtain global threshold to binarize the image. Then, we use morphological closing and opening operations to get the final binary image. The processing results are shown in Figure 3.
(a)
(b)
(c)
2.2. Initial Edge Extraction via Fractional Gaussian Differential
There are two different backgrounds in solder panel images. One is red background and the other is gray background. The solar panel image under red background is used to calculate position shift and deflection angle against the reference position, while gray background image can be used to extract the inner standard lines for measuring edge length of solar panel. For preprocessing, we also perform color space conversion and select subspace for red background image, while for gray one we take subspace from RGB space. Then, in these color subspaces, we all use HPD for image binarization.
After obtaining the binary image, we extract the initial edge of the solar panel by the new fractional calculus [17, 20]. Fractional calculus has been a powerful image enhancement technique in recent years. By adjusting the fractional order of the Gaussian difference, an accurate and antinoise edge extraction effect can be obtained [16, 17].
There are three ways to define fractional calculus, namely, the Grünwald-Letnikov (G-L) definition, the Riemann-Liouville definition, and the Capotu definition. The G-L definition comes from the classic definition of the calculus from integer-order continuous function and generalizes the integer calculus into fractional order. The G-L definition is described as [16]
Thus, fractional partial differential can be defined as ( is fractional differential order)
Based on the above two equations, if combined with Gaussian kernel , the fractional Gaussian filtering function can be naturally deduced as
Then if we set the size of filtering window , the variance , and fractional order , the fractional Gaussian filtering template [19] can be acquired as Tables 1 and 2.
When the Gaussian fractional template is used to extract the initial edge of the solar panel image, the threshold and maximum suppression are also performed as conventional edge detectors do but without performing neighborhood connections. The solar panel image preprocessing results are shown in Figures 4 and 5.
(a)
(b)
(c)
(a)
(b)
(c)
2.3. Solder Joint Image Feature Extraction
Once solder joint image is binarized, the image features for defect identification can be extracted. It should be noted that the extracted image features are capable of reflecting the quality of the solder joint; that is, the image features extracted from the good solder point image and the defective one should be different. According to the principle of machine vision inspection [27], since the area and gravity of the image can indicate whether the shape in the image is uniform or not and the anisotropy and the moment of inertia can reflect the symmetry and overall distribution of the image, through (4)–(8), we calculate the area, the gravity , the anisotropy , and the moment of inertia of the binary image as the extracted image features.
For area, this feature can count all nonzero pixels by
Gravity can reflect the uniformity quality of solder joint.
And anisotropy reflects the symmetry quality of solder joint. Before calculating the anisotropy, we should get the center moment in image region :
Then the anisotropy can be acquired by
Moreover, the moment of inertia can be utilized to describe the whole imaging distribution of solder joint:
These six image features are representative and can describe the quality of solder joint at different views. Some examples of solder joint features extraction are shown in Figure 6.
3. ELM-Based Solder Joint Defect Detection Method
ELM is always treated as a single-hidden layer feedforward neural network (SLFN) [21, 22]. For labeled samples , where and , suppose there are hidden neurons with activation function ; the standard SLFN can approximate these samples with zero error, meaning that there exist , , and such that where is the weight vector connecting the th hidden neuron and the input neurons, is the weight vector connecting the th hidden neuron and the output neurons, and is the threshold of th hidden neuron.
Then if we use matrix form, (9) can be written in short as where
is called the hidden layer output matrix of neural network. In ELM, the input weights and the biases are randomly assigned, and then the output weight can be approximated bywhere is the Moore-Penrose generalized inverse (pseudoinverse) [21] of the hidden layer output matrix .
From mathematical analysis, it is easy to see that ELM can be trained efficiently, and its prediction is fast and robust. In practice, when we apply ELM for solder joint defect detection, we firstly extract the mentioned features from solder joint images and input them to ELM. And the output prediction value is or . Here, the output “−1” indicates that the solder joint is defective, whereas the output “1” means that it is good one. Some defect identification examples are illustrated in Figure 8.
4. MLSR Based Edge Estimation Algorithm
Accurate and robust edge estimation plays a key role in determining the position of the solar panel. Since the initial edges extracted by fractional Gaussian differential are small line segments with noised fluctuations, they should be properly corrected and estimated to be a clear and accurate edge line. Based on moving least square regression (MLSR) [25, 26], we improve the least square fitting used in previous work [14] for more accurate and robust solder panel edge estimation.
If we treat the edge as a position function , then the accurate edge estimation becomes the position regression optimization problem when given initial line segments. Through the base function expansion, the edge regression function can be expressed as where is position regression value for certain position . is the -order complete monomial basis function, is the coefficient of the basis function, and is the number of basis functions. From the point of view of moving least square, the coefficient matrix should be determined by minimizing the weighted least square error measure at . can be defined as where is the discrete point to be fitted within the influence domain of . And is weight function which defines the influence domain of and attributes a weight to each discrete point depending on its position relative to . The weight function is positive and the value should be increased with the decrease of the distance between and .
With the original extracted edge, we can fit an initial edge line by randomly choosing two pixels. Then, based on the distance to the initial edge of the rest pixels, we can determine whether we use them to update the edge line or discard them. The detailed MLSR algorithm is given as follows.
Step 1. Assume that the two initial edge pixels are and . Denote the initial fitted line as ; if , then ; else , where .
Step 2. Compute the distance between a rest edge pixel and the fitted edge line . If ( is the distance threshold), then the pixel can be regarded as a true edge pixel that can be used to update . Save such pixel position in a matrix: , , where is the number of true edge pixels.
Step 3. Repeat Step 2 until all candidate edge pixels are processed; then we acquire the turn estimated edge pixels’ position matrix .
Step 4. According to the matrix , use MLSR algorithm to update the initial edge line and acquire this turn edge line denoted as .
Step 5. If the iteration turn number reaches the setting, then exit and output the final refined edge line ; else let and go to Step 2 to continue.
Once the accurate edge line of solar panel is acquired, we can calculate the position parameters position shift and deflection angle against reference position.
5. Experimental Results and Comparisons
5.1. Solder Joint Defect Detection
In our experiments, 30 PCB solder joint images are captured and each of them has 22 solder joints, which means that totally there are 660 solder joints. Some solder joint examples are shown in Figure 7. For ELM solder defect identification, we take 500 solder joints as the training samples and the remaining 160 ones are regarded as the testing samples.
Then we extract the six image features for each solder joint image and utilize ELM to determine whether the solder joint is defective or not. Some defect detection examples are shown in Figure 8.
Generally speaking, the good solder joint can be treated as a tiny circle that means the anisotropy . However, in practice, the value of for good solder joint is sometimes with small fluctuation around 1. In our experiments, if and , then the extracted region is identified as a good solder joint; otherwise it will be a defective solder joint or not a solder joint.
Given the training set , where , and , . The specific procedure of the ELM solder joint defect identification is given as follows.
Step 1. Choose the activation function as , where , and set the hidden neuron number as .
Step 2. Assign the input weights and biases , , randomly.
Step 3. Utilize the solder joint training images to train the network; that is, calculate the hidden layer output matrix by (11).
Step 4. Estimate the output weight by (13).
Step 5. After training, calculate the output by (10), and then identify whether the input solder joint is defective or not.
For comparison, we also used our previous method [14] and SVM classifier to do solder joint defect detection. In our experiments, we conducted eight tests. The detection performance and the comparison results are shown in Table 3, and the consumption time is also shown in Table 3. As can be seen from the table, our method’s average recognition rate is 98.6%; recognition speed is not up to 5 seconds (in Matlab 2014). Experimental results and comparison show that the proposed ELM-based defect detection scheme is effective in both recognition rate and processing speed.
5.2. Solar Panel Position Determination and Edge Estimation
Before determining the position of the solar panel, we should estimate its edge lines accurately under red and gray backgrounds. For red background solar panel image, we only need to acquire four external edge lines. Suppose that there are four vertexes, , , , and ; the initial edge lines can then be fitted by the four vertexes. After using MLSR algorithm, the final edge lines estimation result is shown in Figure 9. The corresponding line parameters, slope and bias, are shown in Table 4. And the result of least square fitting (LSF) method in [14] is compared in Table 4. From the table, we can see that the position estimated by MLSR is more accurate than that of LSF. And we can also find that the estimated edge lines in solder image are not actually parallel, which may be caused by imaging distortion according to the machine vision principles [27].
For gray background solar panel image, it will need eight vertexes to determine two inner standard lines in the middle part and the four edge lines. Suppose that there are eight vertexes, , , , , , , , and ; after using MLSR algorithm, the final refined edges are shown in Figure 10. Lengths of estimated edge lines are calculated and shown in Table 5. Then we can take the difference between the actual detected edge length and the given length to determine whether the solar panel is qualified or not. Similarly, LSF results are also compared in Table 5. From the table, we can see that the mean square error (MSE) of MLSR estimated lines is less than 0.5 mm. This indicates that the performance of MLSR edge estimation is superior to previous one.
All experiments and comparisons under different background show that the proposed MLSR algorithm is efficient in solar panel position determination and edge lines estimation.
6. Conclusions
In this work, machine learning based methods for solder joint defect detection and solar panel position determination are explored in detail. After image preprocessing, six representative image features are extracted and ELM classification is used for defect identification. The experiments and comparisons show that the proposed feature extraction and ELM classification method are effective. Moreover, MLSR algorithm is proposed to estimate the edge lines of solar panel and then to determine the panels’ position. The practical experiments show that the edge lines in different background can be robustly estimated. And then the position parameters can be accurately determined.
In the future, our further study will focus on using deep learning features for solder joint defect detection and developing more robust regression algorithm to overcome the influence of image distortion.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work is partly supported by the Major and Key Project of Natural Science of Anhui Provincial Department of Education (Grant nos. KJ2015ZD09 and KJ2016A085). It is also supported by Anhui Provincial Natural Science Foundation (Grant no. 1608085MF129).