Abstract
In this paper, concentrated auto encoder (CAE) is proposed for aligning photo spacer (PS) and for local inspection of PS. The CAE method has two characteristics. First, unaligned images can be moved to the same alignment position, which makes it possible to move the measured PS images to the same position in order to directly compare the images. Second, the characteristics of the abnormal PS are maintained even if the PS is aligned by the CAE method. The abnormal PS obtained through CAE has the same alignment as the reference PS and has its abnormal characteristics. The presence or absence of defects and the location of defects were identified without precisely measuring the height of the PS and critical dimension (CD). Also, alignment and defect inspection were performed simultaneously, which shortened the inspection time. Finally, inspection performance parameters and inspection time were analyzed to confirm the validity of the CAE method and were compared with the image similarity comparison methods used for defect inspection.
1. Introduction
Interferometry that uses the light interference phenomenon to measure the height and CD of an object can measure a large area at one time without damaging the object being measured. Because of its advantages of high measurement resolution and speed, interferometry has been utilized in the field of thin-film transistor-liquid-crystal display (TFT-LCD), microelectromechanical systems (MEMS), and semiconductors. Interferometry has been verified as an in-line measuring instrument for manufacturing processes as well as for research and development in the precision part manufacturing industry. In the TFT-LCD manufacturing process, an interferometer measures the height and CD of irregularly shaped objects, such as PS, halftone, and color filter [1–5].
TFT-LCD has a digital display that displays information through very thin liquid crystals, which are most widely used in the display industry. In order to display information, this product uses the polarization property of the polarizer and the change in state of the liquid crystal with liquid-solid property to adjust the amount of light passing through. The TFT-LCD consists of two glass substrates with a color filter and a TFT, a liquid crystal injected between two glass substrates, and a backlight unit. PS is the unit that maintains the gap between the two glass substrates. PS is similar to a bump and typically has a bell-shaped form. The height and size of the PS are important parameters that are measured in the manufacturing process because they determine the amount of the liquid crystal that is injected and determine the degree of deformation caused by the external pressure of the display panel [6, 7].
Interferometry is used for repeated measurements of PS and inspection. A 3D height image and 2D tomographic image can be obtained from interferometry measurements [8–10]. The height of the PS can be measured based on the 3D height image, and the axial CD measurement and the defect of the shape itself can be inspected using the 2D tomographic image. It is difficult to verify the shape defect of the PS based only on the axial CD measurement; CD measurements in various directions cause the inspection time to increase. In general, an image similarity comparison method was used to inspect the shape of the PS defect. Three methods can be used to compare image similarity [11–17]: histogram comparison (HC), template matching (TM), and feature matching (FM) methods. These methods are used to determine whether the normal PS template to be inspected exists in the test image. If there is a similar normal PS template, the similarity is outputted. If the degree of similarity is high, PS is considered to be normal, while if the degree of similarity is low, it is possible to determine that the defective shape is an abnormal PS.
In this paper, the CAE method is proposed, which can be used as a replacement for the existing image similarity comparison method that is used for alignment and defect inspection of PS images acquired from an interferometry. CAE allows measured PS images to be aligned. The shape of the PS was inspected, and the PS defect was checked by comparing the reference PS image and the PS image aligned through the CAE. It is possible to identify defects and their location without precisely measuring the height of the PS and CD with the CAE method, and it is also possible to reduce inspection time by simultaneously aligning and analyzing defects. Finally, the performance of the proposed CAE method was verified by comparing it with the existing image similarity comparison methods.
2. Theoretical Background
2.1. Auto Encoder
The auto encoder is a learning network based on deep learning, which learns functions that make the output value similar to the input value. The auto encoder has two neural networks, as shown in Figure 1. The neural network located at the front is an encoder and the network behind it is a decoder. The encoder extracts the feature of the input data, and the original data are regenerated by using this feature and the decoder. By training the input and output data to be as equal as possible, the auto encoder can extract the feature efficiently [18]. The auto encoder is used to remove and restore noise and image noise and can be used for image classification [19–24]. As the amount of research on sensor data utilization and analysis has increased in recent years, it is also possible to use an auto encoder as a correction technique for sensor data.
A typical auto encoder is an artificial neural network that makes output to be input. The target output is used as an input. The encoder is a function of , which extracts the hidden feature from the input vector . In this equation , is dimension weight matrix and means dimension bias vector. is an element-wise activation function that makes a network deeper by changing input data so that it is nonlinear. A sigmoid function or a rectified linear unit (ReLu) is used as an activation function. The formula of the sigmoid function is shown in equation (1). Feature that is calculated in the hidden layer is converted into vector by the decoder . The auto encoder performs the process of extracting and reconstructing the feature from the input data through the encoder and decoder. In conclusion, the input is converted to through the hidden layer and the final output is reconstructed from . Each input is mapped to the related feature , which is then mapped to a reconstruction , which satisfies . The auto encoder must mathematically minimize the difference between input and output . The difference between the input and the output is termed loss, which is also called the average reconstruction error, and the goal of the auto encoder is to optimize the parameter to minimize this loss. The expression of the loss optimization parameter is shown in equation (2), and loss is calculated with equation (3). Loss function is a scalar function that calculates the difference between input and output and is used as an indicator to evaluate the difference of datasets. Mean squared error and cross entropy error are mainly used as a loss function. The loss function in equation (3) is the most commonly used mean squared error function.
2.2. PS Defect Inspection Using Interferometry
Inspection of the PS defect of the display panel is performed using a 3D shape and a 2D tomographic image obtained from an interferometry. It is possible to measure the height from the 3D shape, and it is possible to measure the axial CD of the PS from the 2D tomographic image [7–9]. If the differences between the height or CD of the measured PS and the reference PS values are larger than the threshold, it is judged to be an abnormal PS. This PS defect inspection method confirms the measured value of height, x and y axis CD, assuming that the shape of the PS is bell shaped. Defects in directions other than the x and y axis cannot be detected using the PS defect inspection method. If these precise measured values are not needed and only the presence of defects is required, the image similarity comparison method can be used. The HC method [11, 12], the TM method [13–15], and the FM method [16, 17] are used in this paper for comparing image similarity. Image similarity comparison methods require a test image and a template image for detection. With image similarity comparison methods, a user can determine if the template image that the user is searching for exists in the test image and how similar it is. The location of the PS within the analysis range of interest through image similarity comparison methods can be accurately determined, and the defectiveness can be determined by comparing the similarity.
The HC method is a simple method used to compare histograms between template and test image. A histogram is a graph that displays the distribution of color information for each pixel in an image. In the histogram shown in Figure 2, the x axis corresponds to the brightness value that ranges from 0 to 255 and the y axis indicates the number of pixels corresponding to each brightness value. PS histogram distributions of normal PS and abnormal PS are shown in Figure 2. The normal PS and abnormal PS have different histograms that can be distinguished by their color information. The HC method has the advantage of being simple and requiring less time for analysis. However, since the color information of the image is compressed and compared, the accuracy of the similarity comparison is low.
(a)
(b)
The TM method compares the color information of each pixel of the template image and the test image. The template image moves in the x and y directions and compares the intensities of the pixels of the test and template images to acquire the closest position and corresponding similarity. Hamming distance is used to calculate similarity. The Hamming distance is used to measure the difference between two types of data, such as characters, bytes, and images, and can be expressed as shown in equation (4). and are the intensities of the template and test image, respectively. is the function for mathematical preprocessing, which is applied to the template and test images. The smaller the Hamming distance, the higher is the similarity with the template image. The TM method is characterized in that the more complex the image to be searched for and the larger the test image, the longer is the inspection time. There are various kinds of TM methods, depending on the function of calculating the Hamming distance. The TM method compared in this paper is the TM-SQDIFF method that uses the square of the image intensity difference. The Hamming distance of TM-SQDIFF is defined as shown in equation (5). and denote the template and the test image, respectively. and is the size of the template image to be searched. The window size of the template image is moved in the test image, and the squares of the intensity difference are calculated as the Hamming distance. This Hamming distance has a small value at the TM position. If the template image perfectly matches the test image, it returns 0. Otherwise, the Hamming distance becomes larger.
The FM method extracts the features contained in the template and test images and compares the matching degree of each feature point. In this paper, the SIFT FM method is used to extract features that are invariant in image size and rotation. SIFT is resistant to scale, illumination, translation, rotation, and occlusion of images. However, the amount of computation required to extract these feature points is considerable. The flow of the SIFT algorithm is shown in Figure 3. In the scale-space extrema detection step, a Gaussian pyramid is generated and a difference of Gaussian (DoG) is calculated in order to extract the pole part as a feature point candidate. Gaussian pyramids consist of filters of various scales. SIFT using a Gaussian pyramid extracts feature points by scale, so it can be recognized even if the scale of the target object is changed. Taylor series is used to extract precise features in the key point localization step. Inaccurate feature points are removed from feature point candidates that were extracted from scale-space extrema detection. In the orientation assignment step, the main direction is assigned to each extracted feature point. Gaussian blurring is applied to the 16 × 16 region around the extracted feature point to calculate the direction and magnitude of the gradient. Since this information contains orientation data, SIFT can also recognize rotated objects. In the key point descriptor step, a descriptor is created for each feature point. The key point descriptor is a spatial histogram of the image gradients that is used in the characterization of the appearance of a key point. As shown in Figure 4, a key point descriptor is created by mixing gradient values around the feature points that were obtained in the previous step. Finally, the key point descriptors of each reference and test image are matched by calculating the hamming distance. Image similarity can be examined through the degree of matching of the corresponding feature points.
3. PS Defect Inspection Using Concentrated Auto Encoder
The auto encoder is trained so that A is the output when image A is the input and B is the output when image B is the input. In the proposed CAE method, training proceeds so that only one image is the output. Additionally, training also proceeds so that the auto encoder is concentrated on one image, as shown in Figure 5. The auto encoder is set to input and output , while CAE is set to for input and for output. Training proceeds so that all the inputs output one and the same output. When training for inspection of PS defect in this paper, the normal PSs that are not aligned are inserted in the input and one normal PS that is the align reference is inserted into the output. All input PSs that were not aligned have the same alignment with the reference PS through CAE.
The detailed network of the CAE is shown in Figure 6. The size of PS images at various locations is used as input for the CAE. The CAE is trained so that the input PS images become the reference PS image of size. The reference PS image is measured so that the PS is at the center of the image, and it becomes the align reference of the PS. A convolution block consisting of a convolution layer and ReLu was used to train the CAE. The convolution layer transforms the size and depth of data and extracts their features. Activation functions, such as ReLu, facilitate the extraction of features through nonlinearity of the data. The CAE network shown in Figure 6 uses seven convolution blocks. The convolution layers that comprise the encoder reduce the size of the image to and increase its depth to 128. The encoder performs training through feature extraction of input data. The decoder in the next part reconstructs the original size of the output image from the extracted features. The convolution layer of the decoder outputs the feature data as the original depth 1, size image. The difference between the final output and reference PS images is termed loss. The Adam optimizer is used to minimize the loss; additionally, the hyperparameter optimization of each convolution layer is possible through CAE training.
Training was performed by setting the image of 10,000 normal PSs that were not aligned as input and one reference PS as output. As the training progresses, the input PS aligns with the reference PS through training, as shown in Figure 7. Figure 7 shows the change in normal PS according to the training epoch. As the number of training events increases, the ability to learn improves so that even small dots on the lower right of the output image can be expressed in detail. This shows that the network that was trained through CAE outputs the same normal PS with the same alignment. When an abnormal PS is inserted into this network, alignment is the same as the reference PS and the abnormal characteristic is maintained, as shown in Figure 8. When a normal PS is the input, the same output as the reference image is the output. If an abnormal PS is the input, the output that differs from the reference image is the output. It is possible to inspect the PS without precisely measuring the PS using the characteristics of the CAE.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
A camera with 880 × 640 pixels and a resolution of 0.1 μm/pixel was used in this experiment. The farthest PSs that were used for training are shown in Figure 9. The distance between PSs is approximately 740 pixels, 74 μm. The trained network can be calibrated up to a distance of 74 μm. Since the reference PS is set to the PS located at the center, it is possible to align the PS 37 μm from the reference PS using the corresponding network, as shown in Figure 10. The training PS set must contain a PS far away from the reference PS and use a deeper network to enhance alignment of the learning ability to a PS further away.
(a)
(b)
(a)
(b)
(c)
(d)
(e)
(f)
4. Experiment and Result
Figure 11 shows the flow of defect inspection using CAE. The first item that the defect inspection flow checks for is the presence of a pretrained network. A pretrained network is created by the preliminary training of input images. In this paper, this pretrained network is used to align PS and to maintain the characteristics of the defects. If there is a pretrained network, defect inspection is possible by inserting a test image into the network. However, if there is no pretrained network, it is necessary to measure the input image with an interferometer and to train the CAE in order to create a network.
A total of 10,000 normal PS images were used for CAE training. PS images that were not aligned were measured at various locations within the camera field of view (FOV). The PSs have a positional difference of up to 74 μm within the camera FOV. PS training images were divided into 8 : 2 ratios and distributed to training and validation PS sets. The PS training set was used for actual training, and the PS validation set was used to evaluate the performance of the network that was trained. Since the validation set is not used for direct training, the loss is larger and the accuracy is lower than the training set, as shown in Figure 12. However, the training and validation sets gradually converged as they approached 100 epochs.
(a)
(b)
The network that trained 100 epochs converged sufficiently and showed stable loss and accurate results. The network was verified to be sufficient for training and validation sets, but its performance has not been verified in test sets comprising new PS images. In order to verify testing performance, five different types of PS images (200 test images of each type) that were not used in training comprised a test set. The loss and accuracy of the test set also converged at approximately 100 epochs, as shown in Figure 13. Although the test set showed numerically lower accuracy as compared to the training and validation sets used in direct training, it can be confirmed that the CAE network can be used even in a test set that was not used for training. Overfitting of the trained CAE network did not occur, and it can be applied to various PS images for defect inspection.
(a)
(b)
Network training of up to 100 epochs maintained the characteristics of the PS and aligned the PS to the center of the image. If the image to be reconstructed is complex and diverse, detailed image restoration is possible by increasing the number of training repetitions or using a deeper network. When a normal PS is inserted into a network in which training was completed, a normal PS is the output. If an abnormal PS is inserted, an abnormal PS having the same alignment as a normal PS is the output. As shown in Figure 14, when abnormal PSs are inserted into the CAE, the PSs with the same alignment and abnormal characteristics are the output.
(a)
(b)
(c)
(d)
The degree of abnormal characteristics of the PS can be understood by performing the difference operation on the aligned abnormal PS and the reference PS. Since PS defect inspection requires judging abnormal products, it is possible to distinguish defect judgements using equation (6). The abnormal PSs that were inspected using these conditions are shown in Figure 15. An abnormal PS has a definite form when proceeding with the difference operation with the reference PS.
(a)
(b)
(c)
(d)
The accuracy of defect inspection can be confirmed by the confusion matrix. Precision, recall, accuracy, and F1-score are the parameters of inspection performance that are found in the confusion matrix. Precision is the ratio of correctly predicted ones that are expected to be normal, recall is the ratio of correctly predicted true to normal, and accuracy is a well-predicted rate of the overall sample. Accuracy shows the total reliability of the corresponding network, while precision and recall reveal the skewness of the network. Since a trade-off relationship exists between precision and recall, the accuracy of the method can also be confirmed using the average of these two values, which is called F1-score. The F1-score is used as an inspection parameter to evaluate performance of the learning network. Calculation of the F1-score is shown in equation (7). Table 1 shows the confusion matrix of the proposed CAE method, and Table 2 shows the result of the inspection performance parameter.
In all cases of methods, the recall values are close to 1, which indicates that all PSs that are actually normal are correctly determined to be normal. Image similarity comparison methods and the CAE method are stable in the normal PS determination. In the HC method, precision and accuracy are lowest since the color information of the abnormal PS is combined by a histogram. The HC method can be used to determine the approximate defect judgement of the PS in a short time but cannot identify the location of the defect in each abnormal PS. TM and FM methods show higher inspection performance than the HC method. Both methods are more accurate than the HC method, but they can only extract the presence or absence of defects like the HC method. This is a limitation of the image similarity comparison methods. In the case of precision, recall, accuracy, and F1-score, the proposed CAE method generated better results. The CAE method can accurately judge defects and at the same time identify the location of defects in the image through subtraction with the test image. Since image similarity comparison methods use the entire template image and compare the similarity without alignment, it is hard to determine the defect of the small part. The CAE method is more advantageous than other methods for defect judgement of the small part since the PS to be analyzed is aligned through the auto encoder and then the similarity is compared.
If such a CAE network was trained in advance, it is possible to find defects by inserting a test image into a network and comparing the output image with a reference. In contrast, image similarity comparison methods compare the degree of similarity with the registered template for each histogram, pixel, or feature of the image. Image similarity comparison methods require a longer analysis time when the size of the test image or template becomes larger. Table 3 shows the inspection times for seven different sizes of test images.
Although the HC method has the shortest inspection time, the accuracy of the HC method is low and is not suitable for use in actual manufacturing processes. The TM method, FM method, and CAE method can be used to inspect defects because of their high inspection accuracy. Among these, the FM method requires more than twice the amount of inspection time as compared to the other methods. Inspection time is an important parameter in the manufacturing process. Considering defect inspection time and accuracy, the TM method and CAE method can be applied to equipment used in the manufacturing process. When image analysis is performed on a smaller image size, the inspection speed of the TM method is faster. In contrast, when large image analysis is performed, the inspection speed of the proposed CAE method is faster than the TM method. With the largest image size of 4000 × 3000, the inspection time of the CAE method was reduced by 79% as compared to the TM method. The inspection area is increased in the manufacturing process, and cameras with adequate resolution are predominately used. In other words, situations that require analysis of images with a large pixel size are frequently encountered; thus, using the proposed CAE method can increase the process yield by shortening the inspection time.
5. Conclusion
This paper proposed the CAE method for inspection of PS defects in the display panel. The CAE method has two characteristics: First, unaligned images can be moved into the same alignment position, which allows for the measured PS images to be moved to the same position so that the images can be directly compared. Second, the characteristics of the abnormal PS are maintained even if the PS is aligned by the CAE method. The abnormal PS obtained through CAE has the same alignment as the reference PS and has its abnormal characteristics. These characteristics of CAE allow for the inspection of defects in the PS. In order to confirm the performance of PS defect inspection using CAE, image similarity comparison methods and other defect inspection methods were compared. The abnormal PS defect inspection was performed through CAE, and the confusion matrix and the F1-score were calculated from the inspection results. Results of this study are summarized below.
The CAE method permits movement of the position of nonaligned PSs. The position of PSs that were 37 μm away from the center was moved to the center of the image using the CAE method in these experiments. Even if the PS was moved, the characteristics of normal and abnormal PSs were maintained; thus, the CAE method can judge defects by comparing the aligned output image and the reference image.
The CAE and image similarity comparison methods are useful for determining a normal PS according to each confusion matrix. The CAE method is more advantageous than the image similarity comparison methods for determining an abnormal PS since the CAE method can detect small defects during abnormal PS inspection. In addition, F1-score comparison confirmed that the CAE method has higher precision and recall values and showed superior results of defect inspection. In particular, the CAE method is more effective since the CAE method can analyze not only the PS defect but also the location of the PS defect.
As with the image similarity comparison methods, the inspection time of the CAE method increased as the size of test images increased. Based on the accuracy of inspection and inspection time, the TM and CAE methods are appropriate for manufacturing process equipment. The inspection time of the CAE method and TM method was most similar with image sizes of 1280 × 1024 and 1920 × 1080. The TM method performed better with smaller image sizes, while performance of the CAE method with large image sizes was better in terms of inspection time. It is useful to use the CAE method in the manufacturing process to reduce inspection time because inspection of large-sized images is frequently performed in order to increase inspection yield.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported in part by the Brain Korea 21 Plus, Institute of Engineering Research, Institute of Advanced Machines and Design at Seoul National University.