Advances in Multimedia

Volume 2015, Article ID 285969, 10 pages

http://dx.doi.org/10.1155/2015/285969

## Video Superresolution Reconstruction Using Iterative Back Projection with Critical-Point Filters Based Image Matching

Department of Communication Engineering, Xiamen University, Xiamen, Fujian 361005, China

Received 27 December 2014; Accepted 23 February 2015

Academic Editor: Constantine Kotropoulos

Copyright © 2015 Yixiong Zhang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### Abstract

To improve the spatial resolution of reconstructed images/videos, this paper proposes a Superresolution (SR) reconstruction algorithm based on iterative back projection. In the proposed algorithm, image matching using critical-point filters (CPF) is employed to improve the accuracy of image registration. First, a sliding window is used to segment the video sequence. CPF based image matching is then performed between frames in the window to obtain pixel-level motion fields. Finally, high-resolution (HR) frames are reconstructed based on the motion fields using iterative back projection (IBP) algorithm. The CPF based registration algorithm can adapt to various types of motions in real video scenes. Experimental results demonstrate that, compared to optical flow based image matching with IBP algorithm, subjective quality improvement and an average PSNR score of 0.53 dB improvement are obtained by the proposed algorithm, when applied to video sequence.

#### 1. Introduction

Since high-resolution (HR) images/videos are important in many applications, such as astronomy, military monitor, medical diagnosis, and remote sensing, superresolution (SR) reconstruction has a great significance in practice [1]. The concept of superresolution (SR) reconstruction refers to reconstructing a high-resolution (HR) image from one or more low-resolution (LR) images. The purpose of superresolution (SR) reconstruction is using digital image processing algorithm to enhance the spatial resolution by transcending the limiting factors of optical imaging system [2, 3]. The innate character of superresolution (SR) reconstruction is using complementary content of multiple images to extend high frequency component.

Most superresolution reconstruction methods contain four steps: registration, map, interpolation, noise, and blur removal. Registration refers to estimating motion vectors between two different video frames or images. Then the motion vectors are used to map the pixels of the input low-resolution frames to a common high-resolution reference frame. Interpolation is used to obtain the pixel value of the superresolution grid, by utilizing the mapped pixels. Finally, noise and blur removal is applied to eliminate the optical sensor blur [4].

A variety of iterative superresolution reconstruction algorithms have been proposed. These algorithms can be divided into two types: frequency domain method and spatial domain method. The frequency domain approach was first proposed by Tsai and Huang [5]. They formulated a series of equations which relate high-resolution frames to low-resolution frames, making use of the shift property of Fourier transform. But their imaging model does not consider motion blur and additive noise and is only restricted to global translational motion. Interpolation based method, iterative back projection (IBP) method [1, 6, 7], projection onto convex set (POCS) method [3, 8], and maximum a posteriori (MAP) method [9] are four main spatial domain algorithms for superresolution reconstruction. Interpolation based method is the simplest spatial domain algorithm, which uses multiple registered images to generate the HR image, based on some interpolation approach, such as nearest-neighbor interpolation, bilinear interpolation, and cubic spline interpolation [10]. The other three spatial domain SR algorithms (IBP, POCS, and MAP) are all based on iterative reconstruction and have better results than interpolation based method. Another class of superresolution reconstruction algorithm is example-based superresolution algorithm. Freeman et al. [11] presented the idea of example-based superresolution reconstruction algorithm. A state-of-the-art algorithm via structure analysis of patches was proposed by Kim et al. in [12].

The key to the spatial domain algorithms is accurate image registration. Traditional registration approaches estimate the global translational or rotational motion. But if more than one type of motion coexists in the scene such as natural video, global motion estimation does not work. In video superresolution, image registration algorithms such as block based matching [13] and optical flow based matching [14, 15] are often used. But both of these two algorithms have some drawbacks and are not appropriate in the superresolution reconstruction. Motion vectors cannot be accurately obtained because of block based matching criterion, and the performance of optical flow based matching may be seriously degraded due to changes of brightness.

In this paper, image matching using multiresolutional critical-point filters (CPF-IM) [16–18] is proposed to be applied in the process of superresolution reconstruction. CPF-IM is suitable for representing both the global and local motions. Moreover, the influence of brightness change is small in CPF-IM. In our experiments, pixel-level motion fields are first obtained by CPF-IM. Then the iterative back projection (IBP) algorithm is employed to reconstruct the high-resolution image based on the motion fields. Experimental result shows that the IBP algorithm with CPF-IM has better performance than bilinear interpolation algorithm, bicubic interpolation algorithm, and optical flow based image matching with IBP algorithm.

The rest of this paper is organized as follows. Section 2 briefly introduces the CPF based image matching algorithm. Section 3 describes the proposed superresolution algorithm using iterative back projection (IBP) algorithm with CPF-IM. Section 4 discusses the experimental results. Finally, Section 5 concludes this paper.

#### 2. Image Matching Using Multiresolutional Critical-Point Filters

Multiresolutional critical-point filters (CPF) [16–18] provide a means of matching two images in pixel-level accurately. Suppose there are two images, a source and a destination. A set of multiresolution subimages are constructed for both images. Then, mappings from the source to the destination subimages are performed at each level from the coarsest resolution to the finest resolution. The mapping is computed pixel by pixel constrained by the inherited and the bijectivity conditions. The mapping with the minimum energy will be selected as final correspondence.

Supposing the width and the height of the original image size are and , respectively, denotes the hierarchy level of the finest resolution. A multiresolution hierarchy of size images can be computed. There are four subimages to be calculated, which are formed by extraction of the minimum, the maximum, and the saddle points, respectively, at each level of the hierarchy. Let denote the pixel at () in the subimage, where is the level of hierarchy and is the type of subimage. The pixels of subimages in the hierarchy are recursively calculated from the pixels of its higher level subimages in the hierarchy as follows:where , which are the pixels of the original image.

Once the multiresolution hierarchy is constructed, a top down method is utilized to map pixels from the source image to the destination image. The number of candidate mappings at each level is constrained by the mapping at its upper level. A pixel at level of the source image is searching for its corresponding pixel in the destination image. Suppose the 4 nearest pixels of the pixel are , , , and . Their parents () are mapped to , , , at level . For each of the parents, one child pixel is selected. The four children pixels define an inherited quadrilateral , inside which we search the pixel with a minimum mapping energy.

Let be the pixel to map in source image at location and the pixel to test in destination image at location . The mapping energy consists of and , defined aswhere is a real number. denotes the intensity differences between the source image pixel and its corresponding pixel in the destination image. Considerwhere the function denotes the intensity of image pixel. And is the cost related to the locations of pixel. Considerwhere is a real number. Consider is determined by the differences between and to prevent a pixel being mapped to a pixel too far away. is determined by the distance between the displacement of and the displacement of its neighbors. This energy is used to smooth the mapping.

In the mapping procedure, the energy of the candidate pixels satisfying the above conditions will be computed and compared. Then the pixel with the minimum energy will be determined as the final corresponding pixel.

The advantage of integrating CPF based image matching into video image superresolution is that pixel-level motion fields can be accurately obtained. Shifts usually vary across small regions between video frames; therefore, block based image matching is not accurate enough. Moreover, optical flow based image matching is seriously affected by brightness change. In summary, CPF based image matching can overcome the defects of traditional image matching algorithms and is applicable to global motion and local motion model. In the process of image matching, one of the low-resolution images is selected as the reference image. CPF based image matching is performed between the reference image and the other low-resolution images to obtain motion fields.

#### 3. Integrate CPF Based Image Matching into Video Image Superresolution

This section introduces how to integrate CPF-IM into video image superresolution. A new framework of iterative back projection algorithm is proposed to reconstruct the high-resolution video image.

##### 3.1. Video Superresolution Reconstruction Model

The acquisition of low-resolution images is shown in Figure 1. The relation between the th observation image and the original high-resolution image can be expressed aswhere represents the affine operation matrix for , denotes the blur operation caused by point spread function (PSF), and is the downsampling matrix.