Superresolution of Images Using Area Preserving Geometric Evolution Laws

Abstract

PDE-based approaches are used to obtain superresolution of images. Using level set method for area constraint geometric evolution laws allows to remove pixelation in images without removing small-scale features. We compare mean curvature flow, surface diffusion, and Willmore flow for this purpose.

1. Introduction

The goal of superresolution is to construct out of a given low-resolution image an image with higher resolution. Superresolution is a common problem in image processing and numerous algorithms have been proposed [1]. Most of these algorithms are based on interpolation and lead to an improvement of the image but also to imperfect reconstructions with false high-frequency components. For interpolation methods based on PDEs, which are mostly adapted from inpainting problems, we refer to [2–6]. These approaches try to reconstruct the geometric properties of the images by evolving their level curves. In order to maintain fidelity to the original low-resolution image, different approaches are used. A standard level set method to modify pixel intensity which is based on evolution of mean curvature flow 𝜕_𝑡𝜙=𝐻‖∇𝜙‖, where 𝜙 is the intensity and 𝐻 the mean curvature, as used for example, in [7, 8] for edge-preserving smoothing, noise removal, and other image enhancement, smoothes away feature objects, as all level curves are shortening and thus will not lead to appropriate superresolution images. In [6], this is circumvented by imposing constraints that preserve accuracy to the original image using unchanging anchor pixels defining explicit topology constraints. In [4], the problem is attacked from a different point of view, by constraining the mean curvature flow problem in order to preserve the area enclosed by each level curve. We will follow this approach and extend it to other geometric evolution problems of higher order. Higher-order PDEs have been applied to various problems in image processing, for example, [9–11]. They are, however, not yet been used for superresolution. Only in [12] a higher-order model based on a Cahn-Hilliard equation is applied to superresolution of binary images, especially text.

The paper is organized as follows: in Section 2, we describe a level set formulation for area conserving mean curvature flow, surface diffusion, and Willmore flow. In Section 3, we discuss an appropriate discretization which conserves all level curves simultaneously. In Section 4, the algorithms are applied to test problems. Thereby we start from a given 256×256 image, lower the resolution, and use bicubic interpolation to create an image that is again 256×256. Our three area preserving evolution laws are applied to this image for a short time and the results are compared with the original image. We also apply the algorithms to text images which are only known at a low resolution. In Section 5, we draw conclusions.

2. Level Set Method

For a given initial curve ℳ₀, a geometric evolution law defines a family of curves ℳ(𝑡), 𝑡≥0 with ℳ(0)=ℳ₀. If ℳ(𝑡) is described implicitly as a specific level curve of a function 𝜙(𝑡), we can evolve ℳ(𝑡) by evolving 𝜙(𝑡) by solving 𝜕_𝑡𝜙+𝑣‖∇𝜙‖=0, with 𝑣 a given normal velocity. If 𝑣 is determined through a geometric evolution law, we can implicitly evolve the curve ℳ(𝑡) by solving this level set equation [13]. To simultaneously evolve all level curves ℳ_𝑐 of a level set function 𝜙, we can define a global energy 𝐸𝜙=_ℝ𝑒ℳ_𝑐𝑑𝑐=_Ω∇𝜙𝑓𝑑𝑥,(1) using the coarea formula. Thereby, we consider 𝑒ℳ_𝑐=_ℳ𝑐𝑓𝑑𝑠(2) with 𝑓 a given energy density on the curve ℳ_𝑐 and an appropriate gradient flow with respect to a specific metric 𝑔(ℳ_𝑐)𝑣=−grad_{𝑔(ℳ𝑐)}𝑒ℳ_𝑐.(3) Following [14], we interpret 𝜙 as an element of the manifold ℒ of level set ensembles and identify a tangent vector 𝑠=𝜕_𝑡𝜙 on ℒ with a motion velocity 𝑣 of the corresponding level curve ℳ_𝑐 via the classical level set equation 𝑠+𝑣‖∇𝜙‖=0. This allows to define the corresponding metric on ℒ which reads for the 𝐿² metric𝑔^𝐿_𝜙𝑠₁,𝑠₂=_ℝℳ𝑐𝑣₁⋅𝑣₂=𝑑𝑠𝑑𝑐_Ω𝑠₁𝑠∇𝜙²=∇𝜙∇𝜙𝑑𝑥_Ω𝑠₁𝑠₂∇𝜙⁻¹𝑑𝑥(4) and for the 𝐻⁻¹ metric𝑔^𝐻−1_𝜙𝑠₁,𝑠₂=_ℝℳ𝑐𝑣₁⋅𝑣₂=𝑑𝑠𝑑𝑐_ΩΔ_ℳ⁻¹𝑠₁𝑠∇𝜙²=∇𝜙∇𝜙𝑑𝑥_Ω𝑠₁𝑠₂∇𝜙⁻¹𝑑𝑥.(5) For both cases, we are now able to rewrite the simultaneous gradient flow of all level curves in terms of the level set function𝜙𝜕_𝑡𝜙=−grad_𝑔𝜙𝐸𝜙,(6) which is equivalent to𝑔_𝜙𝜕_𝑡𝜙,𝜂=−𝐸^′𝜙,𝜂(7) for all functions 𝜂∈𝐶^∞₀(Ω). From (7) the weak form can be obtained for various geometric evolution laws in level set form by choosing appropriate energy density 𝑓, see Table 1 (e.g., [14, 15]).

Table 1 

Construction of geometric evolution laws in level set form, with ℎ the mean curvature.

To define area preserving versions of these geometric evolutions laws, we introduce a Lagrange multiplier 𝜆_𝑐 and define the Lagrangien 𝑙ℳ_𝑐,𝜆_𝑐=𝑒ℳ_𝑐−𝜆_𝑐(area(𝑡)−area(0))(8) with area(𝑡)=_Ω𝑐(𝑡)𝑑𝑥(9) the area of the interior domain Ω_𝑐 inside of ℳ_𝑐. We thus obtain 𝑣=−grad_{𝑔(ℳ𝑐)}𝑙ℳ_𝑐,𝜆_𝑐=−grad_{𝑔(ℳ𝑐)}𝑒ℳ_𝑐+𝜆_𝑐(10) with 𝜆_𝑐 uniquely determined by the constraint ∫_ℳ𝑐𝜕_𝑡𝑥𝑑𝑠=0: 𝜆_𝑐=∫_ℳ𝑐grad_{𝑔(ℳ𝑐)}𝑒ℳ_𝑐𝑑𝑠∫_ℳ𝑐𝑑𝑠.(11) The corresponding evolution laws read as follows:(i)area preserving mean curvature flow 𝑣=ℎ+𝜆_𝑐,𝜆_𝑐=∫_ℳ𝑐ℎ𝑑𝑠∫_ℳ𝑐,𝑑𝑠(12) (ii)surface diffusion (which is already area preserving) 𝑣=Δ_ℳℎ,(13) with Δ_ℳ the surface Laplacian;(iii) area preserving Willmore flow 𝑣=Δ_ℳℎ+ℎ‖𝑆‖²₂−12ℎ²+𝜆_𝑐,𝜆_𝑐∫=−^ℳ𝑐Δ_ℳℎ+ℎ‖𝑆‖²₂−(1/2)ℎ²𝑑𝑠∫_ℳ𝑐,𝑑𝑠(14) thereby 𝑆 denotes the shape operator on ℳ_𝑐 and ‖⋅‖₂ the Frobenius norm.

To construct a level set formulation to simultaneously evolve all level curves ℳ_𝑐 under the constraint, that their enclosed area is preserved, we follow the same lines as for the unconstraint problem and define 𝐿𝜙,𝜆=_ℝ𝑙ℳ_𝑐,𝜆_𝑐𝑑𝑐(15) to obtain 𝜕_𝑡𝜙=−grad_𝑔𝜙𝐿𝜙,𝜆(16) or equivalently𝑔_𝜙𝜕_𝑡𝜙,𝜂=−𝐿^′𝜙,𝜆,𝜂(17) for all functions 𝜂∈𝐶^∞₀(Ω) and 𝐿^′=𝐿_𝜙. For area preserving mean curvature flow (𝑓=1 and 𝑔_𝜙=𝑔^𝐿2_𝜙), we thus obtain_Ω𝜕_𝑡𝜙∇𝜙𝜂𝑑𝑥=_Ωℎ𝜂−𝜆𝜂𝑑𝑥,_Ωℎ𝜂𝑑𝑥=−_Ω∇𝜙𝜆(𝑥)=∫∇𝜙⋅∇𝜂𝑑𝑥,_Ω𝛿𝜙𝑥^′−𝜙(𝑥)ℎ𝑥^′∇𝜙𝑥^′𝑑𝑥^′∫_Ω𝛿𝜙𝑥^′−𝜙(𝑥)∇𝜙𝑥^′𝑑𝑥^′.(18) The same formulation can be obtained by rewriting the formulation in [4] in weak form. For surface diffusion (𝑓=1 and 𝑔_𝜙=𝑔^𝐻−1_𝜙), we obtain_Ω𝜕_𝑡𝜙𝜂𝑑𝑥=_Ω𝑃∇ℎ⋅∇𝜂∇𝜙𝑑𝑥,_Ωℎ𝜂𝑑𝑥=−_Ω∇𝜙∇𝜙⋅∇𝜂𝑑𝑥,(19) with 𝑃=𝐼−∇𝜙/‖∇𝜙‖⊗∇𝜙/‖∇𝜙‖ the projection onto the tangent space 𝒯_𝑥ℳ_𝑐 defined for every 𝑥 on ℳ_𝑐. For area preserving Willmore flow (𝑓=ℎ² and 𝑔_𝜙=𝑔^𝐿2_𝜙), we obtain_Ω𝜕_𝑡𝜙∇𝜙𝜂𝑑𝑥=_Ω𝜇𝜂−𝜆𝜂𝑑𝑥,_Ω𝜇𝜂𝑑𝑥=−_Ω𝜔²2∇𝜙³∇𝜙⋅∇𝜂−𝑃∇𝜔∇𝜙⋅∇𝜂𝑑𝑥,_Ω𝜔∇𝜙𝜂𝑑𝑥=_Ω∇𝜙𝜆(𝑥)=∫∇𝜙⋅∇𝜂𝑑𝑥,_Ω𝛿𝜙𝑥^′−𝜙(𝑥)𝜇𝑥^′∇𝜙𝑥^′𝑑𝑥^′∫_Ω𝛿𝜙𝑥^′−𝜙(𝑥)∇𝜙𝑥^′𝑑𝑥^′,(20) with 𝜔=−‖∇𝜙‖ℎ a weighted curvature.

3. Semi-Implicit Finite Element Discretization

We discretize the constraint evolution problems first in space using piecewise linear finite elements and then in time based on a semi-implicit backward Euler scheme. We thereby follow the discretizations of the unconstrained problems [11, 14] and treat the Lagrange multiplier explicit by evaluating 𝜆_𝑐=∫_Ω𝛿𝜙𝑥^′−𝑐𝛼𝑥^′∇𝜙𝑥^′𝑑𝑥^′∫_Ω𝛿𝜙𝑥^′−𝑐∇𝜙𝑥^′𝑑𝑥^′(21) (with 𝛼=ℎ,𝜇) for different discrete values of 𝑐 and then interpolating to obtain 𝜆, see [4]. The algorithms are implemented in the adaptive finite element toolbox AMDiS [16].

We first demonstrate the smoothing and area preserving properties of the algorithms. We, therefore, evolve a perturbed circle. Thereby the radius of ℳ₀ is initially defined as 𝑟=𝑟₀(1+epscos(freq𝜃)) with angle 𝜃∈[0,2𝜋). Figure 1 shows the initial curve and the evolution of its averaged curvature over time towards the constant value ℎ=1/𝑟₀. We here only use the constraint for ℳ₀ and let all other level lines evolve accordingly, thus only 𝜆₀ is computed and 𝜆=𝜆₀. For all cases, the area remains constant; however, the smoothing differes from which a different behaviour if applied to images is expected.

Figure 1 

Area preserving evolution laws for perturbed circle with 𝑟0=0.5, eps=0.075, and freq=16.0. Different time steps are used for the evolution laws.

4. Numerical Results

We start with a simple application, namely, the superresolution of text images. Therefor, we use the logo of our university and apply area preserving Willmore flow, mean curvature flow, and surface diffusion to smooth the jagged parts of the text given at a low resolution. Figure 2 shows the results and the improvement if compared with bicubic interpolation. The differences obtained with the three evolution laws are minor.

(a)

(b)

(c)

(d)

(e)

Figure 2 

Superresolution of text (a) bicubic interpolation, (b) Willmore flow, (c) mean curvature flow, (d) surfaces diffusion, (e) zoom from left to right: bicubic interpolation, Willmore flowm mean curvature flow, and surface diffusion. The evolution results are shown after 10 time steps. The number of time steps used results from our individual taste for the best result.

The described algorithms can be extended to color images by applying them separately to the individual color planes. As already mentioned in [6], the possible separation of such an approach, due to the independent evolution, is not of significant for the obtained results.

We consider two images, the “Lena’’ image and the “Monarch’’ image. For both images, originally given with a resolution 256×256, the resolution is reduced to 75×75 and bicubic interpolations is used to produce again a 256×256 image. We apply our three level set algorithms to the resulting images. Figure 3 shows results on the “Lena’’ image.

(a)

(b)

(c)

(d)

Figure 3 

“Lena’’ image from left to right: (a) bicubic interpolation, (b) Willmore flow, (c) mean curvature flow, (d) surfaces diffusion. The evolution results are shown after 20 time steps in the case of Willmore flow and surface diffusion and 8 time steps for mean curvature flow. The number of time steps used results from our individual taste for the best result.

Applying the same conserved evolution laws to the “Monarch’’ image results in Figure 4.

(a)

(b)

(c)

(d)

Figure 4 

“Monarch’’ image from left to right: (a) bicubic interpolation, (b) Willmore flow, (c) mean curvature flow, (d) surfaces diffusion. The evolution results are shown after 16 time steps in the case of Willmore flow and surface diffusion and 6 time steps for mean curvature flow. The number of time steps used results from our individual taste for the best result.

In both cases, all area conserved evolution laws result in smoother contours, which is directly visible at the shoulder and the hat in the “Lena’’ image and along the white stripes in the “Monarch’’ image. The jagged parts remaining after bicubic interpolations are removed. Without the area constraint, the evolution would oversmooth the images and significant detail would be lost. The applied constraints help to maintain these details. The difference in the three evolution laws is most significant for mean curvature flow, which reduces the sharpness of the edges. The results for Willmore flow and surface diffusion are comparable.

5. Conclusion

We consider PDE bases approaches to reconstruct geometric properties of images by evolving their level curves. Applying these level set reconstructions to obtain superresolution images had already been shown to lead to improved results, as jagged parts of the image can be smoothed. However, unconstrained evolutions will lead to oversmoothing. The goal thus is to constrain the evolution laws such that visually significant properties remain. A simple attempt in this direction, which does not require image-specific treatments is to constrain the area of each level curve. We have shown that such constrained geometric evolution laws can improve results of other methods for image magnification. The resulting PDEs turn to become nonlocal which requires special treatment. For all cases of area conserved mean curvature flow, surface diffusion and area conserved Willmore flow we use a semi-implicit discretization in which the Lagrange multiplier is treated explicitly which drastically reduced the complexity of the discretization scheme of the higher-order PDEs. A comparison of the three evolution laws shows an improvement if fourth-order PDEs are used. Small-scale details seem to better remain than for the second-order PDE. The difference between area conserved Willmore flow and surface diffusion is only marginally in the considered examples. As surface diffusion is already area conserving and thus computationally less expensive, it should be the method of choice.

Acknowledgment

The authors would like to thank P. Smereka for fruitfull discussions.