Mathematical Problems in Engineering

Mathematical Problems in Engineering / 2020 / Article

Research Article | Open Access

Volume 2020 |Article ID 7095248 | https://doi.org/10.1155/2020/7095248

Jingchun Zhou, Dehuan Zhang, Weishi Zhang, "Multiscale Fusion Method for the Enhancement of Low-Light Underwater Images", Mathematical Problems in Engineering, vol. 2020, Article ID 7095248, 15 pages, 2020. https://doi.org/10.1155/2020/7095248

Multiscale Fusion Method for the Enhancement of Low-Light Underwater Images

Academic Editor: Volodymyr Ponomaryov
Received06 Jul 2020
Accepted20 Aug 2020
Published07 Sep 2020

Abstract

To solve the color cast and low contrast of underwater images caused by the effects of light absorption and scattering, we propose a novel underwater image enhancement method via bi-interval histogram equalization. The proposed method consists of three main parts: color correction, contrast enhancement, and multiscale fusion. First, the color cast is eliminated by automatic white balancing. Then, homomorphic filtering is adopted to decompose the image into high-frequency information and low-frequency information, the high-frequency information is enhanced by the gradient field bi-interval equalization which enhances the contrast and details of the image, and the low-frequency information is disposed via gamma correction for adjusting the exposure. Finally, we adopt a multiscale fusion strategy to fuse the high-frequency information, high-frequency after bi-interval equalization, and low-frequency information based on contrast, saturation, and exposure. Qualitative and quantitative performance evaluations demonstrate that the proposed method can effectively enhance the details and global contrast of the image and achieve better exposedness of the dark areas, which outperforms several state-of-the-art methods.

1. Introduction

Over the past decade, the exploration and protection of marine resources have become a widespread concern of the international community because of the abundant resources of oceans, rivers, and lakes. So, the underwater imaging has become an important research field. Clear underwater images can provide important information for underwater archaeology, resource surveys, and marine life research [14]. But there are three main problems with underwater imaging due to the absorption and scattering of light in underwater environment: color cast, underexposure, and low contrast. In the process of underwater propagation, the underwater image will actually produce serious color cast when the image is above 10 meters due to the different attenuation rates of light of different wavelengths. The underexposure is due to the absorption and scattering of light by water which results in the attenuation in underwater brightness. Due to the scattering of underwater light, the contrast in the underwater image is reduced, and distant objects are blurred.

Recently, many image enhancement methods are used in the underwater image defogged, but the defogged methods based on image enhancement cause the phenomenon of excessive enhancement of bright areas and insufficient exposure of dark areas while enhancing underwater images. This paper adopts multiscale fusion strategy to solve the excessive enhancement caused by the enhancement method while enhancing the details and exposure in the dark areas of the image.

In this paper, our method not only removes the color cast of images (color correction in Figure 1) but also enhances the details of the image (contrast enhancement in Figure 1). In order to solve the low contrast of the underwater image, our method employs homomorphic filtering to enhance the image. However, the detail and contrast of the dark area are insufficient. Then, in order to further enhance the detailed information of the dark area of the image, the gradient field bi-interval equalization enhancement is adopted in the high-frequency information, which enhances the contrast and details while causing excessive enhancement in some areas. To solve the excessive enhancements, we propose to adopt a multiscale fusion strategy based on contrast, saturation, and exposure (since low-frequency information contains fewer image details, which is not suitable for human visual characteristics, the global contrast and saturation of low-frequency information are not considered in our method). The three input images are high-frequency information, high-frequency information after gradient field bi-interval equalization, and low-frequency information after gamma correction. Our method significantly enhances the detail and exposure in dark areas of the image (qualitative and quantitative comparison in Sections 4.3 and 4.4). Besides, the experiments demonstrate that our method significantly increases the number of local feature points of image matching and has practical application value in computer vision processing.

The contributions of our method include the following three points:(1)We propose bi-interval histogram to enhance the detail and contrast of the images.(2)We propose to employ low-frequency information as the input image of multiscale fusion to maintain the exposure in image, which only considers the exposure weight.(3)The experiments demonstrate that our method significantly enhances the detail and exposure in dark areas of the image.

The rest of the paper is structured as follows. The related work will be introduced in Section 2. The proposed model will be introduced in Section 3, including white balance, the process of extracting fusion images with related technologies, and multiscale fusion. In Section 4, we demonstrate the validity of the proposed method by qualitative and quantitative assessments, which can solve the common computer vision problem. Section 5 is the conclusion and discussions.

Many underwater image dehazing methods have been proposed by different scholars in the past decades. The existing underwater defogging technologies can be grouped into five categories, which contain enhancement method based on multiple images [59], image restoration method [1019], image enhancement method [2025], underwater image dehazing method based on deep learning [2628], and the method based on fusion [2933]. The existing underwater image dehazing methods have achieved some success, but there are still some shortcomings. The advantages and disadvantages of the existing five categories are shown in Table 1.


AdvantageDisadvantage

Enhancement method based on multiple images [59]The methods can accurately estimate the transmission or attenuation coefficient from multiple images to restore the image scene.However, the method based on multiple images imposes restrictions on imaging conditions and has limited applicability.

Underwater image restoration method [1019]The methods considered the characteristics of underwater images formation and selective estimation of different wavelengths, which can effectively restore the underwater image.However, the underwater image restoration methods [2025] are not established in many underwater scenes. For example, bright sand in the foreground is mistaken for being close to the camera due to it has a high value in all channels.

Image enhancement method based on single image [2025]The image enhancement methods can significantly enhance the contrast and details of the underwater image.Some image defogging methods based enhancement cannot fully recover the color and scene information of underwater images since the method ignores the relationship between the degradation degree of underwater images and the depth of the scene.

Underwater image dehazing method based on deep learning [2628]The methods can better achieve underwater image defogging with the help of recent deep learning.The underwater images in different scenarios, light conditions and the turbidity of water have different characteristics. It is difficult to devise an underwater image dehazing method based on deep learning for the different lighting conditions of all oceans.

Enhancement method based on fusion [2933]The methods can maintain better exposedness in dark areas, improve overall contrast, and edge sharpness.This method is not effective in restoring color cast in some underwater images.

The first category is the enhancement method based on multiple images, which adopts multiple images to recover scene information. The enhancement method based on multiple images needs to be taken from different angles or different time periods. Schechner and Karpel [5] estimated the transmission of the image with the polarization information of light to restore the visibility of the image. Kopf et al. [6] exploited the deep photosystem to recover the image model to get a clear image. Yamashita et al. [7] and Bryson et al. [8, 9] estimated the attenuation coefficient from different perspectives images to restore the image scene. However, the method based on multiple images imposes restrictions on imaging conditions and has limited applicability.

The second category is the image restoration method. The classic method is the dark channel prior proposed by He et al. [10]. However, dark channel prior has many disadvantages when used in underwater images. Chiang and Chen [11] assumed that the recovered transmission is the transmission of red channels via combining with the characteristics of underwater light selective attenuation. Drews et al. [12] assumed that the main source of underwater visual information is blue and green channels. Perez et al. [13] evaluated DCP changes in underwater autonomous vehicles. Li et al. [14] proposed a novel global background light estimation algorithm and the medium transmission estimation algorithm for the regression model. However, the prior is not true in many underwater scenes. For example, bright sand in the foreground is mistaken for being close to the camera due to the fact that it has a high value in all channels. Besides, the water in background has a dominant color (at least one-color channel is low). However, many methods can not accurately estimate the transmission, which is only suitable for underwater scenes such as shallow water with low backscattering.

Many scholars have proposed some methods of image restoration based on the physical model to overcome these limitations, which estimate the parameters of the model to derive the degradation process. Lu et al. [15] restored underwater image by wavelength compensation. Peng et al. [16] estimated the distance of the scene through the blurriness of the image. Peng and Cosman [17] proposed a method to recover and enhance the underwater images in the image formation model (IFM) based on image blurriness and light absorption. The above methods did consider the characteristics of underwater images formation and selective estimation of different wavelengths but ignored the change of attenuation coefficient with the sensor, ambient light, and other factors. Akkaynak and Treibitz [18] proposed a revised underwater image formation model which considered the backscatter signal and direct transmission signal of underwater. Akkaynak and Treibitz [19] proposed the Sea-thru method based on [18], which used the dark pixels and known distance information to estimate backscatter. However, the algorithm of the revised method is complex and comparatively less applicable.

The third category is image defogging method based on image enhancement. The representative methods include histogram equalization and the method based on Retinex theory. Traditional histogram equalization results in a significant decrease in the contrast of some areas due to combining the pixel value of the small probability in the original image. Huang et al. [20] employed the relative global histogram based on wavelength attenuation to improve the contrast. Fu et al. [21] firstly adopted color correction, then decompose the image into illumination and reflectance with an alternating direction optimization strategy, enhanced illumination and reflectance, respectively, and finally merged the enhanced images. Zhang et al. [22] improved the Retinex theory and combined bilateral filter and trilateral filter based on the characteristics of three channels in the CIELAB color space. Nnolim [23] proposed a partial differential equation-based enhancement for underwater images using fuzzy homomorphic processes. Zhang et al. [24] exploited secondary guided filtering to reduce the noise of image and white balance to eliminate the influence of illumination in images. Zhou et al. [25] employed Laplacian filter to enhance the reflection and performed linear fusion with the illumination after correction to obtain the enhanced image. However, the image defogging method-based enhancement cannot fully recover the color and scene information of underwater images since the method ignores the relationship between the degradation degree of underwater images and the depth of the scene.

The fourth method is underwater image dehazing method based on deep learning. Li et al. [26] proposed WaterGAN, which exploited an unsupervised generative network to generate realistic underwater images from aerial images and depth pairings. Sun et al. [27] employed the pixel-to-pixel (P2P) network to enhance underwater images, the encoder of P2P was composed of three convolutional layers, and the decoder was composed of three deconvolution layers. Uplavikar et al. [28] exploited domain antagonism learning (UIE-DAL) to enhance underwater image which consists of three losses. Although the underwater image dehazing method based on deep learning has obtained certain achievements, the underwater images in different scenarios, light conditions, and the turbidity of water have different characteristics. It is difficult to devise an underwater image dehazing method based on deep learning for the different lighting conditions of all oceans.

The fifth category is image enhancement method based on fusion of input images processed with different operations to obtain images with different features, which are fused by the fusion strategy. Mertens et al. [29] proposed an exposure fusion algorithm based on contrast and saturation. Ancuti et al. [30] proposed an underwater image enhancement method based on global contrast, local contrast, saliency, and exposedness, which preprocessed the input image to generate two fusion images of color correction and contrast enhancement for multiscale fusion. Shen et al. [31] proposed a method based on boosting the Laplace pyramid that fully considered the local exposure, global exposure, and JND-based saliency of the fused image. Ancuti et al. [32] proposed an underwater image enhancement method based on [30], which obtained the color compensation image and the white balance image from the original image based on the global contrast, saliency, and saturation for multiscale fusion to obtain the final image. Yin et al. [33] proposed the improved pulse-coupled neural network, which decomposes source images into high- and low-frequency components, and the pulse-coupled neural network process high-frequency and the sparse representation enhancing the low-frequency; the final image is obtained by the reconstruction of high- and low-frequency components by NSST inverse transformation, which combines neural deep learning and image fusion.

3. The Proposed Model

In this section, our method employs white balance for correction of underwater images, then, we adopt homomorphic filtering to decompose the image after white balance into high-frequency and low-frequency information, and our method uses gradient field bi-interval equalization to further enhance the detail edge of high-frequency information. Then, low-frequency information is processed by gamma correction. Finally, our method multiscale fused the high-frequency information and enhanced high-frequency and low-frequency based on contrast weight, saturation weight, and exposure weight. As shown in Figure 1, our underwater image enhancement method consists of three steps: color correction, contrast enhancement, and multiscale fusion.

3.1. Color Correction

The severe color cast of underwater images is caused due to the light of different wavelengths with different absorption and scattering characteristics when traveling in turbid underwater. The proposed method adopts automatic white balance method [34] to correction color, which mainly includes detection and adjustment of the white point.

The original underwater image is uniformly partitioned into 12 areas in YCbCr color space. The area is ignored when the value of DCb or DCr is close to 0, in which DCb or DCr represents the average absolute differences in the area of Cb channel or Cb channel. Then, other areas determine the white reference point according to equations (1) and (2); the points in the top 10% of the brightness value are the final reference white points:where indicates the pixel value in Cb channel, indicates the pixel value in Cr channel, m represents the width of the area, and n indicates the height of the area. Then, our method converts the image from YCRCB space to RGB space.

Our method adjusts the underwater image by the reference white points:where i indicates the channel in RGB space, represents the average pixel value of white point in the i channel, MaxY is the maximum value of the Y channel in the YCbCr color space, and Wi is the adjusted pixel value.

To effectively illustrate the effectiveness of white balance, the following methods are compared with each other in different scenarios: the classic Max RGB [35], Gray world [36], and the state-of-the-art methods including Shades-of-Gray [37], Gray-Edge [38], and Weighted Gray-Edge [39]. And the different scenes include green, blue, and purple. As clearly seen from Figure 2, the white balance adopted in our method can effectively remove the color cast of the underwater images in different scenes and enhance the visibility of the image.

3.2. Contrast Enhancement

Although the white balance restores the color of underwater images, the image is blurry and of low contrast. The edges and details of the image are blurry due to the fact that the spread of light in water is exponentially attenuated. Therefore, we propose an effective method based on multiscale fusion, which enhances the image into high-frequency and low-frequency information, respectively. Firstly, we decompose the white balanced image into high-frequency and low-frequency information by the homomorphic filtering [40].

The image transforms from spatial domains to frequency domains via using Fast Fourier transformation:where W denotes the output of white balance, i is high-frequency information, r represents low-frequency information, and (u, ) represents the coordinates of the frequency domain.

We adopt Gaussian filters as a transfer function H (u, ) to suppress low-frequency information when enhancing high-frequency information. For the convenience of the next processing, the high-frequency information is converted into the spatial domain via Fast Fourier reverse transformation:where represents high-frequency information and denotes low-frequency information. And Rh, Rl denote the maximum and minimum coefficients values, respectively. D0 is the cut-off frequency and c is the constant that controls the sharpening of the bevel. According to several experiments, it was determined that D0 = 2, Rh = 2.5, Rl = 0.3, c = 3.5.

3.2.1. High-Frequency Information Enhancement

Although homomorphic filtering reduces nonuniform lighting, dark and overexposure areas still exist in high-frequency information. To reduce the dark and overexposure areas of high-frequency information, the gradient field bi-interval histogram equalization is employed to enhance high-frequency information, which improves the overall contrast and brightness of high-frequency information, and further enhances the detail and edge of high-frequency information.

Different images have different image characteristics, but almost all image gradient images have the same characteristics, causing the points to be concentrated in the position of the segmented interval [41]. We calculate the x, y partial derivative of the high-frequency information:where i denotes the R, B, G channel, respectively, Grai is the gradient image of the i channel, and Hom represents the high-frequency information.

To solve the problem that the histogram equalization of gradient image leads many pixels in nonedge region changing to the edge region, which results in enhancement, we proposed bi-interval histogram equalization to avoid the over enhancement. We set suitable threshold of the gradient image divides the gradient histogram into edge region and nonedge region for histogram equalization, respectively. In the gradient histogram, the gradient amplitude of the overall image is approximately subject to the Rice distribution, the gradient amplitude is approximately subject to the mixed Rayleigh distribution in the nonedge region, and the gradient amplitude is approximately subject to the mixed Gaussian distribution in the edge region. According to the distribution of the gradient image histogram, we adopted that the first t value at which the instantaneous skewness equals 0.63 is considered as the adaptive threshold value:where represents the nth-order central moment of the probability density function (PDF) in gradient image.

According to the threshold (t), we firstly divided the histogram of the gradient image into two intervals, which [0, t] (interval I) represents the nonedge region and (interval II) represents the edge region, where represents the maximum in gradient image. Then, we process large and small gradient intervals, respectively,where I1 and I2 represent the original gradient value in the intervals I and II respectively, , denote the resultant gradient value in the intervals I and II respectively, and is the number of pixels in interval II where the gradient value is bigger than t and less than I2. N1 and N2 represent the number of pixels in the intervals I and II, respectively. is the maximum value of the absolute value of . is the gradient regulator, which is usually 1 to 1.5. Our method adopts .

Then, we combine and to I, which represents the target gradient field; then, we establish the target gradient field while maintaining the orientation information of the original gradient field unchanged:

We define that and , where div represents the divergence operator, is the matrix form of DH, A matrix and B matrix are shown in equation (10), and, then, we get equation (11):

Employ the matrix transformation method to solve for ; the detailed steps are as follows:Step 1. The A matrix and B matrix are diagonalized to and , and we obtain P and Q, which is a similar transformation matrix, that is, , .Step 2. Equation (11) can be transformed into , then, we obtain by Step 1.Step 3. The formula from Step 2 is simplified to get .Step 4. can be obtained from Step 3; then X can be calculated, that is, DH.

We adopt the Sobel operator for edge detection to better reflect the effect of the gradient field bi-interval equalization, subjectively indicating the edge detail of the image. As can be seen from Figure 3, the gradient field bi-interval equalization can effectively enhance the brightness and detail of dark areas while enhancing the contrast of the image.

3.2.2. Low-Frequency Information Correction

The homomorphic filtering effectively enhances the detail and edge information, but some areas show excessive enhancement. The gradient field bi-interval equalization eliminates the dark area and exposure area of the high-frequency information, but this causes the overexposure and excessive enhancement phenomenon in other areas at the same time. We propose to employ low-frequency information as a fusion image to solve the above problems.

Some areas in the white balance output are underexposed. If the original low-frequency information is used as a fusion image, it causes the situation that some areas are underexposed and there are fewer details of the image. For more details and solving the underexposure caused by the white balance output, we employ gamma correction [42] to increase the brightness of low-frequency information, while enhancing the details of darker areas of the image. Simple gamma correction is defined aswhere is the low-frequency information, is the maximum pixel value for low-frequency information, and represents the pixel value of low-frequency information after gamma correction.

For better reflecting the importance of low-frequency information after gamma correction as fusion image, in the section of the experiment, we demonstrate the validity by experiment and obtain the parameters according to the experiment.

3.3. Multiscale Fusion

Homomorphic filtering reduces the nonuniform lighting of high-frequency information; although there are still dark areas and low contrast in the image, the details of the brighter areas of the image are more prominent and the gradient field bi-interval equalization improves the overall contrast of high-frequency information, but there is overexposure of some brighter areas. To solve overexposure and overenhancement of high-frequency information, we propose that multiscale fuses the low-frequency information after gamma correction with the high-frequency information and the high-frequency information after gradient field bi-interval equalization.

In this section, multiscale fusion consists of two steps: the definition of the aggregated weight map and the multifusion of input images and aggregated weight map. Aggregated weight map is determined by the three measurement weights, which include the contrast weight, saturation weight, and the exposure weight map [30]. The obvious is that the low-frequency information contains fewer image details and not suitable human visual characteristics, so the global contrast and saturation of low-frequency information are not considered in our method.

3.3.1. Weight Map

(1) Contrast Weight Map. The grayscale image of the input images is used to estimate the global contrast weight WLa with an absolute value to ensure the edge and detail texture information of the image, and the value is large in the edge and detail of the image: where La represents Laplace operator,  denotes convolution, and F is the input image. However, the low-frequency information contains fewer image details, so the proposed method sets the global contrast of low-frequency information to a smaller value, such as 0.001.

(2) Saturation Weight Map. When the image is exposed over a long period, the color becomes unsaturated. For ensuring that the image looks vivid and conforms to the visual characteristics of humans, we employ the standard deviation of each pixel within the channels in RGB color space as the saturation weight WSa: where R (x, y), G (x, y), B (x, y) represent the R, G, B channel of input images, respectively, m (x, y) denotes the average of R, G, B channels in (x, y), and WSa (x, y) indicates the saturation weight of the (x, y). The low-frequency information contains less image color information and not suitable human visual characteristics, so the proposed method sets the saturation of low-frequency information to a smaller value, such as 0.001.

(3) Exposure Weight Map. For solving the problem of overexposure and underexposure in the image, it is necessary to ensure that the pixel value is as close as possible to 0.5, that is, midpoint (to facilitate the fusion of multiple images, the image pixel value is normalized to 0-1). We adopt a Gauss curve with an expectation of 0.5 to represent the exposure weight WE for each pixel point:

in our experiments can get better results [30].

(4). Aggregated weight map. In multiscale fusion, we synthetically consider the contrast, saturation, and exposure of the image that the aggregated weight map is obtained by multiplying the three feature weight maps. We multiply the pixel value of contrast weight map WLa, saturation weight map WSa, and exposure weight map WE corresponding to the pixel point for each input image:

where the subscript z denotes the zth input image, z indicates the number of input images, and Wz is a two-dimensional weight map.

To ensure image consistency, we naturalize the weight map WK:where represents aggregated weight map.

3.3.2. Fusion

The proposed method adopts multiscale fusion to combine the advantages of the image. Besides, the seam effect of the image in areas with rapid weight changes can be minimized via the reconstruction of the Laplacian pyramid. The detailed steps are as follows.Step 1. The input image I is decomposed by Laplacian pyramid, which is defined as . The aggregated weight map is decomposed by Gaussian pyramid, which is defined as where l superscript represents the lth level.Step 2. The Laplacian pyramid and Gaussian pyramid are fused pixel by pixel according to , where represents the Laplacian pyramid of the fusion image.Step 3. The Laplacian pyramid is reconstructed to obtain the fusion image.

As shown in Figure 4, the overexposure phenomenon caused by homomorphic filtering and gradient field bi-interval equalization can be effectively resolved through multiscale fusion of the high-frequency information, the gradient field bi-interval equalization output, and the gamma-corrected output, while retaining its detail information. Compared with the original image, the contrast and details texture of the image have been significantly improved.

4. Experiment Results

This section demonstrates the significance of low-frequency information as fusion image and the effect of gamma correction parameter setting on the experiments; meanwhile, it comprehensively verifies the low-frequency information as fusion image introduced in Section 3.3. Then, based on the qualitative and quantitative evaluation of various data sets, the proposed method is compared with the state-of-the-art underwater images recovery and enhancement techniques. Finally, the SIFT is adopted for local feature point match to demonstrate the effectiveness of our method in image application.

4.1. Verify the Effectiveness of Low-Frequency Information

In Section 3.3, we amply described the importance of low-frequency information as the input image. In the multiscale fusion, low-frequency information can effectively avoid overexposure and underexposure when improving the contrast and details information of the image. We test different images on various datasets in the fusion process; the low-frequency information as input image can availably improve the details of the image. We only provide an obvious effect of the image as a better proof due to limited space.

We set up a set of comparative experiments, including images without low-frequency information and images with low-frequency information in multiscale fusion. From Figure 5, what can be obviously seen is that multiscale fusion without low-frequency information causes overexposure. In the red frame area of g1, obvious light spots are generated, which make the details blurry and affect the contrast of the image. Therefore, low-frequency information compensates exposure. The image details are enhanced with low-frequency information in multiscale fusion.

4.2. Determining Gamma Correction Parameters

The multiscale fusion with low-frequency information can effectively avoid overexposure, but overcompensation phenomenon is generated, which results in the fact that some areas are too dark, and dark spots are generated compared with the surrounding areas. The reasons for overcompensation are as follows: the low- and high-frequency information are obtained from the white balanced image. It can be clearly seen that the white balanced image is underexposure overall, which results in the situation that the low-frequency information is underexposure from Figure 5 b1. With the low-frequency information without processing as the fusion input image, it will lead to overcompensation in the exposure area. The low-frequency information with gamma correction can avoid overcompensation to obtain that image has better contrast and more details.

In Figure 6, it can be obviously seen that when the parameter of gamma correction is 1.25, the overall pixel value of low-frequency information is reduced. Although the detail of the brighter area of low-frequency information is stretched, the dark spot is intensified and overexposure compensation is generated. The overall pixel value of low-frequency information is improved by setting , the details of the dark area of low-frequency information are stretched, and the global contrast is improved, but some areas still are underexposure. The black spot is effectively solved by setting , and the light spot and black spot are balanced; meanwhile, the image details are retained. When , the overall pixel value of low-frequency information is excessively increased due to the parameter being too small, which causes the light spot and the loss of image details.

4.3. Qualitative Comparison

We tested real underwater images by amateurs and underwater images datasets [11, 30, 43], which include rock, fish, and shipwreck. Figure 7 shows the results of ten images. We compared the proposed method with the state-of-the-art methods, which include DCP [10], UDCP [12], IBLA [17], ULAP [44], and RGHS [20]. We can clearly see that all the methods achieved an excellent defogging effect, but some methods are still insufficient. It can be seen from Figure 7 that the DCP method based on outdoor defogging enhances the contrast and details of the image, but it incompletely removes fog from underwater images due to the absorption and scattering that exists in underwater scene. Although UDCP better restored image visibility and highlighted details compared to DCP, the result is with obvious bluish because it ignores the influence of red channel and only estimates underwater transmittance image through the blue and green channel without considering the difference between the three channels transmittance.


DCPUDCPIBLARGHSULAPOURS
PCQIUIQMPCQIUIQMPCQIUIQMPCQIUIQMPCQIUIQMPCQIUIQM

Rock10.7711.0770.5271.7610.9701.2420.7751.2270.8541.4991.0141.547
Rock20.7691.3880.7041.7580.9231.3841.0381.4831.0201.5111.0961.661
Fish10.9181.0210.6601.3130.6841.3781.0091.0910.9841.0071.0941.106
Fish20.7790.7060.7031.0410.9760.9250.9560.8120.9670.9140.9901.033
Fish31.0320.9260.5661.5651.1401.1071.1141.1751.1041.3131.2501.372
Shipwreck10.9041.1780.6121.5600.8411.4270.9711.2881.0391.2251.1581.320
Shipwreck20.7330.8690.5981.5340.7800.9760.9571.1080.8321.1501.0291.489
Shipwreck31.0130.8510.7211.6061.0470.9311.0760.9081.0261.2391.1061.224
Average0.8651.0020.6361.5170.9201.1710.9871.1370.9781.2321.0921.344

IBLA and ULAP methods based on image restoration compared with DCP and UDCP can effectively recover the color of image and, at the same time, effectively enhance the image contrast (as can be observed in Figure 7 Shipwreck1). However, some images show low contrast in IBLA since it relies too much on attenuation ratios between color channels. And some images show the color bias in ULAP, for example, the ULAP method in the Rock2 and Fish1 show the red overall color phenomenon. RGHS method based on underwater image enhancement effectively enhanced the details of image, but the enhanced underwater images still show color cast and cannot restore the color of objects in Rock1 and Fish2 due to the relationship between the degradation degree of the underwater images and the depth of the scene was not considered.

The proposed method effectively removes the color cast in extreme underwater scenes, such as the excellently restoration of the color of moss in Shipwreck2, which has higher robustness than comparative methods. At the same time, our method significantly improves the contrast and object details of images. For example, the proposed method effectively enhances the details information of rocks in Rock2, while maintaining the texture and details of fish and enhancing the details of fish while enhancing the details of plants in the lower right corner of Fish1.

Overall, the proposed method generally achieves better contrast, removes the color cast of underwater images, and effectively restores the color of the object, meanwhile significantly improving image contrast and details information, which can enhance the details in the dark areas while ensuring details in the light areas.

4.4. Quantitative Comparison

We employ two metrics due to avoiding the deviation from qualitative comparison: UIQM [32] and PCQI [32]. We also calculate the average metrics [45] of all the test images to ensure the rationality of the test results. UIQM [32] and PCQI [32] are the common underwater images quality evaluation metric recently. UIQM is an unreferenced underwater color image quality evaluation method, inspired by the human visual system; UIQM evaluates the quality of underwater images by colorfulness (UICM), sharpness (UISM), and contrast (UIConM): . The larger the value of UIQM, the better the balance of image colorfulness, sharpness, and contrast and the better the effect of image enhancement and restoration. We also adopt local area block contrast quality based full reference (PCQI) to evaluate the contrast changes of image. The higher the value of PCQI, the higher the contrast and the clearer the details. The quantitative comparison can be shown in Table 2 and Figures 8 and and 9.

The average of PCQI and UIQM of our method is higher than comparative methods as shown in Table 2. Although the UIQM of the proposed method is lower than UDCP, it can be clearly observed from the qualitative comparison that UDCP shows serious color distortion.

Overall, the proposed method can effectively enhance the details and contrast of underwater images through qualitative comparison and quantitative comparison.

4.5. Application

We can observe that our method has achieved prominent achievement on contrast and color reproduction by Table 2. So, we adopt SIFT [32] to match local feature points, compare the local feature points calculation, and match with the original underwater images to prove that our method can significantly improve the matching rate of feature points.

The set of original images on the left in Figure 10 obtains three valid match points using the SIFT operator, and 41 valid match points are obtained from the enhanced image after our method. Comparing the set of images on the right in Figure 10, the original image obtained four valid match points through the SIFT operator, and when applied to the image obtained by our enhanced method, 50 match points are obtained (there is an invalid match point). Our method can significantly increase the number of local feature points of image matching by improving the contrast of the image and restoring the color of the image. Since the quantitative metrics of the original image are included, two nonreference evaluation metrics A and B are selected.

5. Conclusions and Discussions

We proposed an underwater image enhancement method based on fusion of image different information, which includes high-frequency information, high-frequency information after gradient field bi-interval equalization, and low-frequency information after gamma correction. Our method can restore the color of important objects in underwater images (different cameras, depth, and lighting conditions) of various scenes, while significantly enhancing the contrast and details of the image and effectively improving the details in the dark areas of the image.

The experiments prove that our method can acquire better achievement in both subjective and objective comparison; meanwhile, our method can improve the number of local feature points matching between multiviewpoint images. So, our method has practical value of computer vision processing. However, there are limitations of our method: the color cast of underwater images still exists, especially the color of the sea in the distance area of the image. Thus, there is still some room for improvement with regard to the color correction method.

Data Availability

The data used to support the findings of this study have been deposited in the Baidu cloud disk repository (https://pan.baidu.com/s/1YEPttDjfeWQ3Qf2yVy6GXw; verification code:stux).

Conflicts of Interest

The authors declare no conflicts of interest.

Authors’ Contributions

J.Z. and D.Z. conceptualized the study. J.Z. and D.Z. were responsible for methodology. W.Z. provided software. J.Z., W.Z., and D.Z. validated the study. J.Z. performed formal analysis. W.Z. provided resources. D.Z. performed data curation. J.Z. and D.Z. wrote the original draft. J.Z. visualized the study. W.Z. supervised the study. J.Z. was responsible for project administration. W.Z. was responsible for funding acquisition. All the authors have read and agreed to the published version of the manuscript.

Acknowledgments

This research was funded by the National Natural Science Foundation of China, Grant no. 61702074; Liaoning Provincial Natural Science Foundation of China, Grant no. 20170520196; and Fundamental Research Funds for the Central Universities, Grant nos. 3132019205 and 3132019354.

References

  1. J. Park, D. K. Han, and H. Ko, “Adaptive weighted multi-discriminator CycleGAN for underwater image enhancement,” Journal of Marine Science and Engineering, vol. 7, no. 7, p. 200, 2019. View at: Publisher Site | Google Scholar
  2. M. Mangeruga, M. Cozza, and F. Bruno, “Evaluation of underwater image enhancement algorithms under different environmental conditions,” Journal of Marine Science and Engineering, vol. 6, no. 1, p. 10, 2018. View at: Publisher Site | Google Scholar
  3. A. Gomez Chavez, A. Ranieri, D. Chiarella, E. Zereik, A. Babić, and A. Birk, “CADDY underwater stereo-vision dataset for human-robot interaction (HRI) in the context of diver activities,” Journal of Marine Science and Engineering, vol. 7, no. 1, p. 16, 2019. View at: Publisher Site | Google Scholar
  4. W. Zhang, L. Dong, X. Pan, P. Zou, L. Qin, and W. Xu, “A survey of restoration and enhancement for underwater images,” IEEE Access, vol. 7, pp. 182259–182279, 2019. View at: Publisher Site | Google Scholar
  5. Y. Y. Schechner and N. Karpel, “Clear underwater vision,” IEEE CVPR, 2004. View at: Google Scholar
  6. J. Kopf, B. Neubert, B. Chen et al., “Deep photo,” ACM Transactions on Graphics, vol. 27, no. 5, pp. 1–10, 2008. View at: Publisher Site | Google Scholar
  7. A. Yamashita, M. Fujii, and T. Kaneko, “Color registration of underwater images for underwater sensing with consideration of light attenuation,” IEEE International Conference on Robotics and Automation, pp. 4570–4575, 2007. View at: Google Scholar
  8. M. Bryson, M. Johnson-Roberson, O. Pizarro, and S. B. Williams, “Colour-consistent structure-from-motion models using underwater imagery,” Robotics: Science and Systems, vol. VIII, p. 33, 2013. View at: Google Scholar
  9. M. Bryson, M. Johnson-Roberson, O. Pizarro, and S. B. Williams, “True color correction of autonomous underwater vehicle imagery,” Journal of Field Robotics, vol. 33, no. 6, pp. 853–874, 2016. View at: Publisher Site | Google Scholar
  10. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 12, pp. 2341–2353, 2010. View at: Google Scholar
  11. J. Y. Chiang and Y. C. Chen, “Underwater image enhancement by wavelength compensation and dehazing,” IEEE Transactions on Image Processing, vol. 21, no. 4, pp. 1756–1769, 2011. View at: Google Scholar
  12. P. Drews, E. Nascimento, F. Moraes, S. Botelho, and M. Campos, “Transmission estimation in underwater single images,” IEEE International Conference on Computer Vision Workshops, pp. 825–830, 2013. View at: Google Scholar
  13. J. Perez, P. J. Sanz, M. Bryson, and S. B. Williams, “A benchmarking study on single image dehazing techniques for underwater autonomous vehicles,” in Proceedings of the OCEANS 2017-Aberdeen, pp. 1–9, Aberdeen, UK, June 2017. View at: Google Scholar
  14. C. Li, J. Guo, C. Guo, R. Cong, and J. Gong, “A hybrid method for underwater image correction,” Pattern Recognition Letters, vol. 94, pp. 62–67, 2017. View at: Publisher Site | Google Scholar
  15. H. Lu, Y. Li, L. Zhang, and S. Serikawa, “Contrast enhancement for images in turbid water,” Journal of the Optical Society of America A, vol. 32, no. 5, pp. 886–893, 2015. View at: Publisher Site | Google Scholar
  16. Y. T. Peng, X. Zhao, and P. C. Cosman, “Single underwater image enhancement using depth estimation based on blurriness,” in Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 4952–4956, Hong Kong, China, September 2015. View at: Google Scholar
  17. Y.-T. Peng and P. C. Cosman, “Underwater image restoration based on image blurriness and light absorption,” IEEE Transactions on Image Processing, vol. 26, no. 4, pp. 1579–1594, 2017. View at: Publisher Site | Google Scholar
  18. D. Akkaynak and T. Treibitz, “A revised underwater image formation model,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 6723–6732, 2018. View at: Google Scholar
  19. D. Akkaynak and T. Treibitz, “Sea-thru: a method for removing water from underwater images,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 1682–1691, 2019. View at: Google Scholar
  20. D. Huang, Y. Wang, W. Song, J. Sequeira, and S. Mavromatis, “Shallow-water image enhancement using relative global histogram stretching based on adaptive parameter acquisition,” in Proceedings of the International Conference on Multimedia Modeling, pp. 453–465, Daejeon, South Korea, January 2018. View at: Google Scholar
  21. X. Fu, P. Zhuang, Y. Huang, Y. Liao, X. P. Zhang, and X. Ding, “A retinex-based enhancing approach for single underwater image,” IEEE International Conference on Image Processing (ICIP), pp. 4572–4576, 2014. View at: Google Scholar
  22. S. Zhang, T. Wang, J. Dong, and H. Yu, “Underwater image enhancement via extended multi-scale Retinex,” Neurocomputing, vol. 245, pp. 1–9, 2017. View at: Publisher Site | Google Scholar
  23. U. A. Nnolim, “Improved partial differential equation-based enhancement for underwater images using local-global contrast operators and fuzzy homomorphic processes,” IET Image Processing, vol. 11, no. 11, pp. 1059–1067, 2017. View at: Publisher Site | Google Scholar
  24. W. Zhang, L. Dong, X. Pan, J. Zhou, L. Qin, and W. Xu, “Single image defogging based on multi-channel convolutional MSRCR,” IEEE Access, vol. 7, pp. 72492–72504, 2019. View at: Publisher Site | Google Scholar
  25. J. Zhou, D. Zhang, P. Zou, W. Zhang, and W. Zhang, “Retinex-based laplacian pyramid method for image defogging,” IEEE Access, vol. 7, pp. 122459–122472, 2019. View at: Publisher Site | Google Scholar
  26. J. Li, K. A. Skinner, R. M. Eustice, and M. WaterG. A. N. Johnson-Roberson, “Unsupervised generative network to enable real-time color correction of monocular underwater images,” IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 387–394, 2017. View at: Google Scholar
  27. X. Sun, L. Liu, Q. Li, J. Dong, E. Lima, and R. Yin, “Deep pixel-to-pixel network for underwater image enhancement and restoration,” IET Image Processing, vol. 13, no. 3, pp. 469–474, 2018. View at: Google Scholar
  28. P. Uplavikar, Z. Wu, and Z. Wang, “All-in-one underwater image enhancement using domain-adversarial learning,” 2019, http://arxiv.org/abs/1905.13342. View at: Google Scholar
  29. T. Mertens, J. Kautz, and F. Van Reeth, “Exposure fusion: a simple and practical alternative to high dynamic range photography,” in Proceedings of the Computer graphics forum, vol. 28, no. 1, pp. 161–171, San Diego, CA, USA, August 2009. View at: Google Scholar
  30. C. Ancuti, C. O. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater images and videos by fusion,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 81–88, 2012. View at: Google Scholar
  31. J. Shen, Y. Zhao, S. Yan, and X. Li, “Exposure fusion using boosting Laplacian pyramid,” IEEE Transactions on Cybernetics, vol. 44, no. 9, pp. 1579–1590, 2014. View at: Publisher Site | Google Scholar
  32. C. O. Ancuti, C. Ancuti, C. De Vleeschouwer, and P. Bekaert, “Color balance and fusion for underwater image enhancement,” IEEE Transactions on Image Processing, vol. 27, no. 1, pp. 379–393, 2017. View at: Google Scholar
  33. L. Yin, M. Zheng, G. Qi, Z. Zhu, F. Jin, and J. Sim, “A novel image fusion framework based on sparse representation and pulse coupled neural network,” IEEE Access, vol. 7, pp. 98290–98305, 2019. View at: Publisher Site | Google Scholar
  34. C. C. Weng, H. Chen, and C. S. Fuh, “A novel automatic white balance method for digital still cameras,” IEEE International Symposium on Circuits and Systems, pp. 3801–3804, 2005. View at: Google Scholar
  35. E. H. Land, “The retinex theory of color vision,” Scientific American, vol. 237, no. 6, pp. 108–128, 1977. View at: Publisher Site | Google Scholar
  36. G. Buchsbaum, “A spatial processor model for object colour perception,” Journal of the Franklin Institute, vol. 310, no. 1, pp. 1–26, 1980. View at: Publisher Site | Google Scholar
  37. G. D. Finlayson and E. Trezzi, “Shades of gray and colour constancy. Color and Imaging Conference,” Society for Imaging Science and Technology, vol. 2004, no. 1, pp. 37–41, 2004. View at: Google Scholar
  38. J. Van De Weijer, T. Gevers, and A. Gijsenij, “Edge-based color constancy,” IEEE Transactions on Image Processing, vol. 16, no. 9, pp. 2207–2214, 2007. View at: Publisher Site | Google Scholar
  39. A. Gijsenij, T. Gevers, and J. Van De Weijer, “Improving color constancy by photometric edge weighting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 5, pp. 918–929, 2011. View at: Google Scholar
  40. L. Xiao, C. Li, Z. Wu, and T. Wang, “An enhancement method for X-ray image via fuzzy noise removal and homomorphic filtering,” Neurocomputing, vol. 195, pp. 56–64, 2016. View at: Publisher Site | Google Scholar
  41. D. Sen and S. K. Pal, “Gradient histogram: thresholding in a region of interest for edge detection,” Image and Vision Computing, vol. 28, no. 4, pp. 677–695, 2010. View at: Publisher Site | Google Scholar
  42. S. C. Huang, F. C. Cheng, and Y. S. Chiu, “Efficient contrast enhancement using adaptive gamma correction with weighting distribution,” IEEE Transactions on Image Processing, vol. 22, no. 3, pp. 1032–1041, 2012. View at: Google Scholar
  43. C. Li, C. Guo, W. Ren et al., “An underwater image enhancement benchmark dataset and beyond,” IEEE Transactions on Image Processing, vol. 29, pp. 4376–4389, 2019. View at: Google Scholar
  44. W. Song, Y. Wang, D. Huang, and D. Tjondronegoro, “A rapid scene depth estimation model based on underwater light attenuation prior for underwater image restoration,” in Proceedings of the Pacific Rim Conference on Multimedia, pp. 678–688, Nanjing, China, December 2018. View at: Google Scholar
  45. W. Ren, S. Liu, L. Ma et al., “Low-light image enhancement via a deep hybrid network,” IEEE Transactions on Image Processing, vol. 28, no. 9, pp. 4364–4375, 2019. View at: Publisher Site | Google Scholar

Copyright © 2020 Jingchun Zhou et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


More related articles

 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder
Views686
Downloads449
Citations

Related articles

Article of the Year Award: Outstanding research contributions of 2020, as selected by our Chief Editors. Read the winning articles.