Abstract
A new online visual inspection technique is proposed, based on a wavelet reconstruction scheme over images obtained from the hull. This type of visual inspection to detect defects in hull surfaces is commonly carried out at shipyards by human inspectors before the hull repair task starts. We propose the use of Shannon entropy for automatic selection of the band for image reconstruction which provides a low decomposition level, thus avoiding excessive degradation of the image, allowing more precise defect segmentation. The proposed method here is capable of on-line assisting to a robotic system to perform grit blasting operations over damage areas of ship hulls. This solution allows a reliable and cost-effective operation for hull grit spot blasting. A prototype of the automated blasting system has been developed and tested in the Spanish NAVANTIA shipyards.
1. Introduction
Main ships’ maintenance care consists of periodical (every 4-5 years) hull treatment which includes blasting works; blasting consists in projecting a high-pressure jet of abrasive matter (typically water or grit) onto a surface to remove adherences or rust traces. The object of this task is to maintain hull integrity, guarantee navigational safety conditions, and assure that the surface offers little resistance to the water in order to reduce fuel consumption. That object can be achieved by grit blasting [1] or ultra-high pressure water jetting [2]. In most cases these techniques are applied using manual or semiautomated procedures with the help of robotized devices [3]. In either case defects are detected by means of human operators; this is therefore a subjective task and hence vulnerable to cumulative operator fatigue and highly dependent on the experience of the personnel performing the task. Figure 1 shows a view of ship’s hulls under repair at NAVANTIA’s shipyards.
(a) Shaped hull
(b) Flat hull
(c) Damaged portions
(d) Detail visual appearance
From an operational point of view, there are two working modes: full blasting and spot blasting. Full blasting consists of blasting the entire hull of the ship, while spot blasting consists of blasting numerous isolated areas where corrosion has been observed. Spot blasting is the most demanded operation due to cost saving reasons. This second working mode demands very precise information about position, size, and shape of damaged portions of the hull to make robotic devices [3–5] to achieve maximum efficiency.
This paper proposes a computer vision algorithm which equips a machine vision system (see Figure 2), capable for precisely detecting defects in ship hulls which is simple enough to be implemented in such a way as to meet the real-time requirements for the application.
Because of the textured appearance of the hull’s surface under inspection (see Figures 1(c) and 1(d)), we have used the wavelet transform, and the developed computer vision algorithm includes an image reconstruction approach based on automatic selection of the optimal wavelet transform resolution level, using Shannon entropy.
2. Defect Detection in Textured Surfaces
Texture is a very important characteristic when identifying defects or flaws, as it provides important information for defect detection. In fact, the task of detecting defects has been largely seen as a texture analysis problem. Figure 3 shows several texture images from ship hull surfaces.
In his review Xie [6] classified texture analysis techniques for defect detection in four categories: statistical approaches, structural approaches, filter based approaches, and model based approaches. In his review of fabric defect detection similarly Kumar [7] classified the proposed solutions into three categories: statistical, spectral, and model based. In his review of automated defect detection in fabric, Ngan et al. [8] classified defect detection techniques in textured fabric into nonmotif-based and motif-based approaches. The motif-based approach [9] uses the symmetry property of motifs to calculate the energy of moving subtraction and its variance among different motifs. Many defect detection methods usually use clustering techniques which are mainly based on texture feature extraction and texture classifications. These features are collated using methods such as cooccurrence matrix [10], Fourier transform [11], Gabor transform [12], or the wavelet transform [13].
Spectral-approach methods for texture analysis characterize the frequency contents of a texture image—Fourier transform—or provide spatial-frequency analysis—Gabor filters, wavelets. A two-dimensional spectrum of a visual texture frequently contains information about the periodicity and directionality of the texture pattern. For example, a texture with a coarse appearance analysed from the spectral point of view shows high-frequency components, while a texture with a fine appearance shows low-frequency components. The analytical methods based on Fourier transform show good results in texture patterns with high regularity and/or directionality, but they are limited by a lack of spatial localization. In this field, Gabor filters provide better spatial localization, although their utility in natural textures is limited because there is no single filter resolution that can localize a structure. The wavelet transform has some advantages over the Gabor transform, such as the fact that the variation of the spatial resolution makes it possible to represent the textures in the appropriate scale, as well as to choose from a wide range of wavelet functions.
3. The Wavelet Transform for Defect Detection
The suitability of wavelet transforms for use in image analysis is well established: a representation in terms of the frequency content of local regions over a range of scales provides an ideal framework for the analysis of image features, which in general are of different size and can often be characterised by their frequency domain properties [14]. This makes the wavelet transform an attractive option when attempting defect detection in textured products, as reported by Truchetet and Laligant [15] in his review of industrial applications of wavelet-based image processing. He reported different uses of wavelet analysis in successful machine vision applications: detecting defects for manufacturing applications for the production of furniture, textiles, integrated circuits, and so forth, from their wavelet transformation and vector quantization-related properties of the associated wavelet coefficients; printing defect identification and classification (applied to printed decoration and tampon printed images) by analysing the fractal properties of a textured image; online inspection of the loom under construction using a specific class of the 2D discrete wavelet transform (DWT) called the multiscale wavelet representation with the objectives of attenuating the background texture and accentuating the defects; online fabric inspection device performing an independent component analysis on a subband decomposition provided by a 2-level DWT in order to increase the defect detection rate.
The review of the literature shows two categories of defect detection methods based on wavelet transform. The first category includes direct thresholding methods [10, 11], whose design is based on the fact that texture background can be attenuated by the wavelet decomposition. If we remove the texture pattern from real texture, it is feasible to use existing defect detecting techniques for nontexture images, such as thresholding techniques [16]. Textural features extracted from wavelet-decomposed images are another category which is widely used for defect detection [17, 18]. Features extracted from the texture patterns are used as feature vectors to feed a classifier (Bayer, Euclidean distance, Neural Networks, or Support Vector Machines), which has unavoidable drawbacks when dealing with the vast image data obtained during inspection tasks. For instance, proximity-based methods tend to be computationally expensive and there is no straightforward way of defining a meaningful stopping criterion for data fusion (or division). Often, the learning-based classifiers need to be trained by the nondefect features, which is a troublesome and usually time consuming procedure, thus limiting its real-time applications [10]. For this reason we have focused on direct thresholding methods. The use of direct thresholding presents a main challenge: how to select the decomposition level. On the other hand, direct thresholding presents two main drawbacks: (1) an excessive wavelet decomposition level produces a fusion of defects with the texture pattern and (2) a wrong reconstruction scheme produces false positives when defects are detected.
The work presented here is based on the authors’ research on previous works [10, 19] and addresses abovementioned drawbacks by a new approach based on Shannon Entropy calculation. Its main contribution is the formulation of a novel use of the normalized Shannon Entropy, calculated on the different detail subimages, to determine the optimal decomposition level in textures with low directionality. For this purpose we propose to calculate the optimal decomposition level as the maximum of the ratio between the entropy of the approximation subimage and the total entropy, as the sum of entropies calculated for every subimage.
3.1. Wavelet Decomposition
For an image of size pixels, each level of wavelet decomposition is obtained by applying two filters: a low-pass filter () and a high-pass filter (). The different combinations of these filters produce four images that are here denoted with the subscripts LL, LH, HL, and HH. In the first decomposition level four subimages or bands are produced: one smooth image, also called approximation, , that represents an approximation of the original image and three detail subimages , , and , which represent the horizontal, vertical and diagonal details, respectively. With this notation, represents the original image, , and represent the approximation image in the decomposition level . From each decomposition level we obtain four subimages, designated here as , and , which together form the decomposition level . The pyramid algorithm to obtain , and , as well as the calculation of the inverse transform, can be found in [20]. We will designate as the inverse wavelet transform of a subimage from the resolution level to level 0.
Figure 4 shows the wavelet decomposition for three conveniently scaled levels () of a statistical texture pattern—painted surface—with corrosion defects; the different subimages or bands are shown (named LL, LH, HL, and HH). These were obtained after applying the different coefficients of the wavelet filters. More specifically, the image in Figure 4(a) was decomposed through the application of the Haar wavelet with two coefficients. At level , images of size pixels are obtained by iterative application of the pyramid algorithm. Note also that the subimages corresponding to the different decomposition levels are produced by successively applying the low-pass and high-pass filters and reducing the rows and columns by a factor of two.
(a)
(b)
(c)
(d)
4. Entropy-Based Method for the Automatic Selection of the Wavelet Decomposition Level
In image processing, entropy has been used by many authors as part of the algorithmic development procedure. There are examples of the use of entropy in the programming of thresholding algorithms [21] and image segmenting [22] as a descriptor for texture classification [23]; as one of the parameters selected by Haralick et al. for application to gray level concurrence matrixes and used for texture characterization [24]; as an element in characteristic vector groups used for classification by Bayesian techniques [25], neuronal networks [26], compact support vectors [27], and so forth.
4.1. Automatic Selection of the Appropriate Decomposition Level
In this work we propose a novel approach for the automatic selection of the appropriate decomposition level by means of Shannon entropy. The entropy function was used to identify the resolution level that provides the most information about defects in real textures. For this purpose, the intensity levels of the subimages of the wavelet transform were considered as random samples. The concept of information entropy—Shannon entropy—describes how much randomness (or uncertainty) there is in a signal or an image; in other words, how much information is provided by the signal or image. In terms of physics, the greater the information entropy of the image is, the higher its quality will be [28].
Figure 5 shows how the texture pattern degrades as the decomposition level increases. This degradation is distributed among the different decomposition levels depending on the texture nature and can be quantified by means of the Shannon entropy.
(a)
(b)
(c)
(d)
(e)
The Shannon entropy function [28, 29] is calculated according to the expression where is a set of random variables with outcomes and is the probability of occurrence associated with .
For a 256-gray-level image of size pixels, we define a set of random variables as the number of pixels in the image that have gray level . The probability of this random variable is calculated as the number of occurrences, , divided by the total number of pixels,
To calculate the value of the Shannon entropy on the approximation subimage () and on the horizontal, vertical and diagonal detail subimages , and ) in each decomposition level , we obtain first the inverse wavelet transform of every subimage and then we apply (3)
The normalized entropy of each subimage, for a decomposition level , has been calculated as where is the number of pixels at each decomposition level . Table 1 shows the values for Shannon entropy calculated for images of Figure 5.
Shannon entropy brings us information about the amount of texture pattern that remains after every decomposition level. Considering (2), entropy provides a measurement of the histogram distribution; the higher the entropy the greater the histogram uniformity; that is, a greater amount of texture pattern is contained in the image. As the decomposition level increases, the texture pattern is being removed; that is, the information content decreases; so the histogram distribution gains uniformity. An optimal reconstruction scheme would eliminate the texture pattern, without loss of defect information. To determine this optimal decomposition level we use a ratio (see (5)) between the entropy of the approximation subimage and the sum of the entropies for all detail subimages, so indicates how much information about the texture pattern is contained in decomposition level . Variations in this ratio allow detecting changes in the amount of information about the texture pattern between two consecutive decomposition levels
The goal is to find the optimal decomposition level which provides the maximum variation among two consecutive values because this indicates that, in decomposition level , the texture pattern still present in level has been removed, keeping useful information (defects).
For this purpose we define as the difference between two consecutive values (see (6)). The optimal decomposition level is calculated as the value of for which takes a maximum value. This maximum value points out the greatest variation of information content among two consecutive decomposition levels, which means that both decomposition levels are sufficiently separated in terms of texture pattern information content, and the decomposition process should end. For decomposition levels , indicates that significant texture pattern information still remains in the approximation subimage, and the decomposition process should continue. For decomposition levels ; indicates that the approximation subimage is oversmoothed, and the reconstruction result from such smooth approximation subimage will cause defect loss
Table 2 shows values for coefficients at every image decomposition level () for the different textures shown in Figure 5, together with the values.
Once the optimal decomposition level is obtained, the process ends with the production of the reconstructed image using (7)
4.2. Smoothing Mask
To remove the noise running through the successive decomposition levels, we applied average-based smoothing over image to obtain as shown in (8) where is the size of the smoothing mask (see Figure 6).
5. Results
5.1. Algorithm Implementation
The proposed computer vision algorithm was implemented as shown in the Pseudocode 1, using the C++ programming language. The mother wavelet used for decomposition was the Haar base function with two coefficients, applied up to a fourth decomposition level. A decomposition level higher than four produced the fusion between defects and background, thus reducing the probability of defect detection.
| ||||||||||||||
5.2. Implementation of the Computer Vision System
The computer vision system for visual inspection of ship hull surfaces (Figure 2) has been implemented on a Pentium computer with a Meteor II/1394 card. This card is connected to the microprocessor via a PCI bus and is used as a frame-grabber. For that purpose the card had a processing node based on the TMS320C80 DSP from Texas Instruments and the Matrox NOA ASIC. In addition, the card had a firewire input/output bus (IEEE 1394) which enables it to control a half-inch digital colour camera (15 fps, 1024 × 768 square pixel) equipped with a wide-angle lens (f 4,2 mm).
The software development environment used to implement the system software modules was the Visual C++ programming language powered by the Matrox Imaging Library v9.0. The system also had a Siemens CP5611 card which acted as a PROFIBUS-DP interface for connection with the corresponding robotized blasting system. A Honeywell sensor was used to measure the distance to the ship by ultrasound, with a range of 200–2000 mm and an output of 4–20 mA. User access to the computer vision system was by means of an industrial PDS (Mobic T8 from Siemens) and a wireless access point. Among other functions, the software that has been developed allows the operator to (1) enter the system configuration parameters, (2) visualize the detected areas to blast for validation by the operator before blasting commences, and (3) calibrate the computer vision system.
5.3. Validation Environment
The proposed computer vision algorithm was assessed at the NAVANTIA shipyard in Ferrol (Spain) on a robotized system used for automatic spot blasting. This operation accounts for 70% of all cleaning work carried out at that shipyard. The robotized system (Figure 7) consists of a mechanical structure divided into two parts: primary and secondary. The primary structure holds the secondary structure (XYZ table), which supports the cleaning head and the computer vision system. More information regarding this system can be found in [5].
With the help of this platform, 260 images of ship hulls’ surfaces (with and without defects) were taken, similar to those shown in Figure 3. In this way a catalogue was compiled of typical surface defects as they appear before grit blasting.
5.4. Metrics
To conduct a quantitative analysis of the quality of the proposed segmentation method, we need to use the best suited metrics to that purpose. The performance of image segmentation methods has been assessed by such authors as Zhang [30] and Sezgin and Sankur [16]. They proposed various different metrics for measurement of the quality of the segmentation in a given method, using parameters like position of the pixels, area, edges, and so forth. Out of these, one of the quantitative appraisal methods proposed by Sezgin was selected and examined: Misclassification Error (ME).
ME represents the percentage of the background pixels that are incorrectly allocated to the object (i.e., to the foreground) or vice versa
The error can be calculated by means of (9), where (background pattern) and (object pattern) represent the pattern image of the background and of the object taken as reference, and (background test) and (object test) represent the image to be assessed. In the event that the test image coincides with the pattern image, the classification error will be zero and therefore the performance of the segmentation will be the maximum.
The performance of the implemented algorithms is assessed according to the equation:
5.5. Algorithm Appraisal
The proposed visual inspection algorithm (see Pseudocode 1) was applied to the above mentioned catalogue that had been taken at the shipyard (some samples are shown in column (a) of Figure 8). The Shannon entropy was calculated and normalized for four wavelet decomposition levels and the optimal level was calculated (6). Images were also processed applying algorithms proposed by Han and Shi [10] and Tsai and Chiang [19]. The result was 3 sets of 260 reconstructed images in which the defects have been isolated from texture. To check the quality of the defect detection algorithms we have concluded with a binarization stage. For that purpose we have selected Kapur’s method [21] which belongs to the group of entropy-based methods, as classified by Sezgin and Sankur [16] in his review of thresholding methods; this has resulted in 3 sets of 260 images (column (b) of Figure 8 shows some results obtained with the proposed algorithm; column (c) of Figure 8 shows some results obtained with the Tsai algorithm and column (d) of Figure 8 shows some results obtained with Han algorithm).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
To apply the metrics described above, human inspectors were needed to segment each of the catalogue images manually (samples of these are shown in column (v) of Figure 8). Table 3 shows the performance () when sample texture images of Figure 8 were segmented using the three algorithms.
As can be observed from above results, the proposed entropy-based algorithm achieved better results than Tsai algorithm and significantly better results than Han algorithm. In both cases the proposed algorithm obtains higher performance with low decomposition level.
We have also analysed the behaviour of the proposed algorithm as misclassification rates. A set of 120 images were processed by the proposed algorithm and also by Han and Tsai algorithms. Results were then analysed by a skilled blasting operator, who assessed what portions of the shown hull surface would be blasted in real conditions at the repair yard. Table 4 shows the average number of defect points classified as Type I and Type II errors for 120 samples of the 260-image set indicated above.
As we can see, the proposed algorithm produced better results as regards false positives—that is, points marked as defective when they are not (Type I error). This is essentially because the operator tends to blast larger areas than necessary, and moreover he is less able to control the cut-off of the grit jet. On the other hand, the proposed algorithm identified similar false negatives (Type II error). This difference was not very significant and is quite acceptable in view of the clear advantage offered by the computer vision system equipped with the proposed inspection algorithm as regards Type I errors.
6. Conclusions
This paper has presented a computer vision algorithm based on the wavelet transform which brings a robust method for detecting defects in ship hull surfaces. To achieve this, we used an image reconstruction approach based on automatic selection of the optimal wavelet transform resolution level by means of a novel use of the Shannon entropy, calculated on the different detail subimages.
The algorithm has been incorporated to a computer vision system that masters a robotized system for blasting ship hulls, making it possible to fully automate grit blasting operation. The results as regards reliability were very similar to those achieved with human workers, while faster inspection was provided (among 8% for flat surfaces in oil tankers and 15% for shaped hulls like frigates) and the consequences of operator fatigue minimized.
Acknowledgments
The work submitted here was carried out as a part of the projects ViSel-TR (ref. TIN2012-39279) and EXPLORE (ref. TIN2009-08572) funded by the Spanish National R&D&I Plan.