Single image super-resolution (SISR) is a traditional image restoration problem. Given an image with low resolution (LR), the task of SISR is to find the homologous high-resolution (HR) image. As an ill-posed problem, there are works for SISR problem from different points of view. Recently, deep learning has shown its amazing performance in different image processing tasks. There are works for image super-resolution based on convolutional neural network (CNN). In this paper, we propose an adaptive residual channel attention network for image super-resolution. We first analyze the limitation of residual connection structure and propose an adaptive design for suitable feature fusion. Besides the adaptive connection, channel attention is proposed to adjust the importance distribution among different channels. A novel adaptive residual channel attention block (ARCB) is proposed in this paper with channel attention and adaptive connection. Then, a simple but effective upscale block design is proposed for different scales. We build our adaptive residual channel attention network (ARCN) with proposed ARCBs and upscale block. Experimental results show that our network could not only achieve better PSNR/SSIM performances on several testing benchmarks but also recover structural textures more effectively.

1. Introduction

Super-resolution (SR) is an important issue in the image restoration area. The task of single image super-resolution (SISR) is to find high-resolution (HR) images from the low-resolution (LR) images. Since it is an ill-posed problem, there are potential high-resolution images corresponding to an identical image with low resolution. There are practical applications with SISR methods, such as video quality enhancement, remote sensing image processing, and MRI analysis. To find the most suitable HR images, there are various methods for SISR problem and other image restoration tasks.

Deep learning has shown its amazing performance in various tasks [15]. Nowadays, there are convolutional neural network- (CNN-) based works focusing on SISR problem. As far as we know, SRCNN [6] is the first work using a three-layer CNN for image super-resolution. After SRCNN, Dong et al. proposed FSRCNN [7] with a deeper but narrower network and achieved better performance with less time cost. Usually, when the network is deeper, the performance will be better. VDSR [8] proposed by Kim et al. used a very deep network design with global residual learning. Inspired by VDSR and residual connections, EDSR [9] proposed by Lim et al. applied an enhanced deep network for SISR problem with residual blocks. Besides EDSR, MDSR [9] designs an upscale module for different scaling factors jointly. Motivated by the structure of Laplacian pyramid, Lai et al. proposed LapSRN [10] with a progressive structure. Similar to MDSR, the progressive LapSRN could upscale images with different scaling factors concurrently. Recursive structures could effectively enlarge the receptive fields and make full use of sharing parameters. DRCN [11] proposed by Ghifary et al. used shared convolution layers to increase the receptive fields. DRRN [12] proposed by Tai et al. combined residual and recursive structures and achieved good performances.

ResNet [13] proposed by He et al. has been proved to be a success for network design. In ResNet, a residual block was proposed for image classification and beyond with effective gradient flow. There are works for super-resolution with residual blocks. EDSR stacked residual blocks with a global shortcut to build a very deep network. RDN [14] proposed by Zhang et al. introduced a residual dense block (RDB) with feature fusion, achieving good performance. Besides RDN, RCAN [15] proposed by Zhang et al. designed a residual-in-residual structure to build the network deeper. There is a shortcut in residual block: the addition of feature maps before and after processing is regarded as the final result. In fact, the above methods apply a fixed ratio balancing the two parts, which does not distinguish the different importance.

Attention is used for human brain simulation. When watching pictures, human’s brain usually focuses on more important area. There are attention methods for image processing tasks. SENet [16] proposed by Hu et al. introduced a channel attention method for importance distribution among channels. SENet requires few parameters, which is flexible for different network designs. To the best of our knowledge, RCAN [15] is the first image super-resolution work with channel attention mechanism. After RCAN, IMDN [17] proposed by Hui et al. modified the vanilla channel attention layers and achieved good performances with few parameters. SAN [18] proposed by Dai et al. introduced a second-order attention mechanism with channel and nonlocal attentions.

In this paper, we propose a novel adaptive residual channel attention block (ARCB) for image super-resolution. Different from vanilla residual blocks, an adaptive weight is learned from paired data for combining information of main path and shortcut. Considering the different importance of channels in residual blocks, channel attention is introduced in ARCB for weight distribution on channels. Besides block designs, recent works design special upscale modules for different scaling factors. In this paper, we introduce a simple but effective general upscale block design for different factors. The adaptive residual channel attention network (ARCN) is designed based on ARCBs and proposed upscale block. Experiments are performed on several testing benchmarks. The results show that our ARCN could not only achieve better performance on PSNR/SSIM comparison but also recover complex structural textures more effectively.

The contributions of this paper could be demonstrated as follows:(1)We propose a novel block named ARCB with channel attention mechanism. In ARCB, we propose an adaptive residual connection with learned weights. The weight factors could find suitable ratios for combination information from different paths. Channel attention mechanism in ARCB distributes different weights on channels for concentrating more on important information.(2)We propose a tiny but effective upscale block design method. With the proposed design, our network could be flexibly analogized for different scaling factors.(3)Experimental results show that our proposed ARCN achieves better PSNR/SSIM results on several testing benchmarks and recovers more complex structural textures than other methods.

2.1. Single Image Super-Resolution

Let and denote LR and HR images separately; the observation model of degradation step could be described aswhere denotes the degradation and n denotes the noise. Usually the degradation models are chosen as bicubic-down with different scaling factors. Given , the task of single image super-resolution (SISR) is to find corresponding . However, there are several potential HR images degraded to a same LR image. Since SISR is an ill-posed problem, it is challenging to find the solution.

Convolutional neural network (CNN) has been proved as an effective tool for image restoration [15]. Recently, there are CNN-based works for SISR problem [19, 20]. To the best of our knowledge, SRCNN [6] proposed by Dong et al. is the first deep learning work for SISR. There are three convolution layers in SRCNN, corresponding to the three steps of sparse coding method: feature extraction, nonlinear mapping, and restoration. After SRCNN, FSRCNN [7] proposed by Dong et al. applied a deeper network for SISR problem. Different from SRCNN, FSRCNN used a deconvolution layer to upscale the feature map. By using shrinking and expanding modules, FSRCNN decreased the number of parameters and built the network deeper and faster. ESPCN [21] proposed by Shi et al. introduced a pixel shuffle layer to substitute the deconvolution layer for upscaling. Similar to FSRCNN, there are several convolution layers processing the LR feature maps. At the end of ESPCN, a convolution layer changed the channel number of feature maps, and the pixel shuffle layer performed the spatial transformation. From the practical experience, a deeper network usually results in better performance. VDSR [8] proposed by Kim et al. applied a very deep network with twenty convolution layers and global residual connection. Besides, batch normalization (BN) was used in VDSR to improve the performance. To preserve the resolution of feature maps, zero padding strategy was applied in VDSR. RED [22] proposed by Mao et al. introduced a symmetrical encoder-decoder structure with convolution and deconvolution layers. To transmit information to the bottom layer, residual connections were designed between blocks.

A deeper network will cause a large amount of parameters. Recursive design with sharing parameters is one of the choices to build lightweight networks. There are recursive networks for SISR problem. DRCN [11] proposed by Ghifary et al. applied sharing convolution layers to enlarge the receptive field with limited parameters. Similar to SRCNN, there are three modules in DRCN. Embedding network extracted the feature maps from input images. After feature extraction, the inference network with sharing parameters performed the nonlinear mapping. Finally, reconstruction network restored the HR images from feature maps. To increase the network performance, there is a skip connection in DRCN to learn the residual information. DRRN [12] proposed by Tai et al. designed a recursive residual connection with sharing parameters to prevent gradient disappearance. By using recursive design and sharing parameters, DRRN built a 52-layer network with few parameters and performed better than VDSR.

Recently, there are works with good performances focusing on different block designs and network pipelines [2327]. Dense connection has been proved effective for image super-resolution [23, 25, 26]. SRDenseNet [28] proposed by Tong et al. introduced dense connection to SISR problem and concatenated all feature maps as the final output. There are four components in SRDenseNet. Firstly, one convolution layer was used to extract low-level features. After extraction, several dense blocks were used to extract high-level features. Deconvolution layers were used to upscale the feature maps. Finally, a convolution layer was applied to HR image production. SRResNet [29] used residual blocks with skip connection to build a deep network for better performance. EDSR [9] proposed by Lim et al. removed redundant batch normalization layers from SRResNet, building a deeper network. Residual connection has been proved as an effective design for better network performance. RDN [14] proposed by Zhang et al. introduced a residual dense block named RDB combining two kinds of structures. By stacking RDBs with global feature fusion, the deep network RDN achieved good performance. MSRN [30] proposed by Li et al. introduced a multiscale residual block combining residual and inception blocks. Global fusion structure was applied in MSRN for feature extraction. To build a deeper network, a novel residual-in-residual structure was proposed in RCAN [15], which turned out to be a success.

2.2. Attention Mechanism

Attention mechanism was firstly proposed by human brain simulation. When watching an image or reading a sentence, the important areas will be paid more attention. There are different attention methods used in image processing. There are four kinds of attention mechanisms: item-wise soft attention, item-wise hard attention, location-wise soft attention, and location-wise hard attention. The difference between item-wise attention and location-wise attention is input form. Special sequential items are required for item-wise attention, while location-wise attention needs a single feature map. From another point of view, attentions could be separated as soft and hard attention. Soft attention focuses more on different areas and channels. After training, soft attentions will be generated by networks. Besides soft attention, hard attention concentrates more on different pixels. Hard attention is a random prediction procedure, which is usually implemented by reinforcement learning.

Spatial transformer network (STN) [31] is an attention method in spatial domain. In STN, information from origin images was transformed into another space with keypoints. The authors proposed a spatial transformer module for the transformation. There are also works for channel domain attention. SENet [16] proposed by Hu et al. introduced a channel attention method for concentrating more on important channels. In SENet, a squeeze-and-excitation (SE) module is proposed to automatically learn the importance of different channels. In SE module, squeeze operation is firstly introduced to get the global channel features. After squeezing, excitation module is used to learn the relations among channels. There are two full connection layers with a ReLU activation in excitation module. Finally, scale module was applied after excitation for reweighting the feature maps. SENet focuses on the importation of channels, regarding different areas of feature maps equally. To consider the global information of feature maps, nonlocal neural networks [32] introduced a long-range dependencies attention block for better performance. However, the proposed nonlocal blocks required more memory cost and high computation complexity.

3. Method

In this section, we will describe the proposed ARCN. In ARCN, an adaptive residual channel attention block named ARCB is proposed to compose the network. Adaptive factors in ARCB for different information importance are learned while training. After adaptive residual connection, channel attention mechanism distributes the weights on channels, which considers the importance from another point of view. The main body of ARCN is composed of several ARCBs and a padding structure. A global skip connection is introduced to ARCN for residual learning. After the main body, an effective and tiny upscale module is designed for changing the scaling factors flexibly. We will introduce proposed ARCN in the following manner: Firstly, the network design will be described in general. After description, the details of ARCB will be discussed with channel attention. Detailed introduction of flexible upscale block will follow the description of ARCB. Finally, some comparisons will be done with other SISR works.

3.1. Network Design

The entire network structure is shown in Figure 1.

There are three modules in the proposed ARCN. Firstly, feature extraction module extracts feature maps from input LR images. After feature extraction, nonlinear mapping module processes the feature maps from LR space into HR space. A skip connection is applied to nonlinear mapping module for global residual learning. Finally, restoration module with a flexible upscale block restores the HR images from proposed feature maps.

There is one convolution layer in feature extraction module. The layer extracts low-level features from LR image and builds the feature maps. Let be the feature extraction module; then the operation could be demonstrated aswhere denotes the input LR images.

After feature extraction, several ARCBs are applied in nonlinear mapping modules for mapping feature maps from LR space to HR space. Let us denote as the output of k-th ARCB; then there iswhere denotes the operation of k-th ARCB.

After K blocks, there is a padding structure composed of two convolution layers with ReLU activation. The padding structure is used to increase the network depth and weight the information from main path for global residual learning. The operation of padding structure and global residual learning could be demonstrated aswhere denotes the padding structure and HR denotes the feature maps after padding.

Finally, an effective upscale block is applied in restoration module. In restoration module, the final HR image is restored from processed feature maps. The operation of restoration module could be demonstrated aswhere denotes the upscale block.

3.2. Adaptive Residual Connection Block

ARCB is introduced to the network with adaptive residual connection and channel attention. An illustration of proposed ARCB is shown in Figure 2(b).

There are two convolution layers with ReLU activation in ARCB. Different from ResBlock, which is used in most of SISR works, a channel attention layer is designed after the convolution layers. The attention layer weights information from different channels. After that, learned adaptive factor is used to scale the processed feature maps.

Let and denote the main processing path and attention layer, respectively; then the operation of k-th ARCB could be demonstrated aswhere denotes the adaptive factor learned while training.

The adaptive factor is one of the main differences between ResBlock and ARCB. In vanilla ResBlock, the ratio of information mixture from two paths is fixed. However, it does not distinguish the importance of different information. In the proposed ARCB, weight factor is a learnable parameter. In other words, the ratio will be adjusted due to the training data, which is more suitable for information fusion.

Another main difference between vanilla ResBlock and ARCB is the channel attention. Convolution layers treat information from different channels equally. To concentrate more on important channels, channel attention is introduced to ARCB. The structure of channel attention is shown in Figure 3.

From Figure 3, global average pooling is firstly applied to information estimation. There is a hypothesis that when the feature maps are more complex, the information will be more important. From this point of view, the global average pooling operation could extract the information fast and effectively. After the feature extraction, a squeeze-and-excitation design is introduced for nonlinear mapping. In squeezing step, the channel number shrinks for information distillation. The most important information will be weighted after squeezing. Then, the excitation module preserves the channel number the same as the origin feature maps. Finally, a Sigmoid activation and a dot multiplication are adopted to distribute different importance among channels.

Let us denote by and the input and output of channel attention. The operation of channel attention mechanism could be demonstrated aswhere denotes the Sigmoid activation; and denote the squeeze and excitation modules. There is a ReLU activation between squeeze and excitation modules. The two modules are made of full connection layers.

There is a Sigmoid activation between the excitation and multiplication. On one hand, it will be helpful for nonlinearity. On the other hand, Sigmoid activation will convert weights to be negative. Since there is no negative for human vision system, it is designed to fit the biological process.

3.3. Flexible Upscale Block

Upscale block is widely used in various works for SISR problem, which increases the resolution of feature maps and restores the final HR image. There are different upscale block designs for different scaling factors without a unified pattern. In this paper, we proposed a flexible upscale block design pattern. With the proposed design, the structure could be easily modified for different scaling factors. The structure of our proposed flexible upscale blocks is shown in Figure 4.

As shown in Figure 4, there are three traditional upscale block designs: A, B, and C [30]. In A, a nonlinear upscale design is introduced with ReLU activation. Design B removes the ReLU activations but keeps the convolution and pixel-shuffle layers. Besides, design C uses deconvolution to substitute the pixel-shuffle layers. Different form other works, there is only one convolution layer with a pixel-shuffle layer in the proposed upscale block.

There are two benefits of using the flexible upscale block design. On one hand, there is only one convolution layer in the block, which saves the parameters and decreases the computation complexity. On the other hand, when the scaling factor is changed, the only modification of the block is the channel number of convolution layers. After changing, the main body of the network could be find-tuned for a new factor with few iterations.

There is a main difference between the proposed block and others. In other designs, there is a convolution layer after the last pixel-shuffle or deconvolution layer. Usually, it is used to restore the HR images with 3 channels from feature maps. However, in our proposed block, the restoration is proposed by the only convolution layer. On one hand, it is corresponding to the feature extraction module, which is also composed of only one convolution layer.

To introduce the design in detail, there are examples of different scaling factors. The special configurations are shown in Table 1. Notice that when the scaling factor is changed, the only modification is the channel number. From this point of view, the proposed upscale block is flexible for different factors.

4. Discussion

(1)Difference from DRDN [33]: In DRDN, there are dense block (DB) structures for feature exploitation. The entire DRDN holds a global residual dense connection design to efficiently process the features. In ARCN, global and local residual connections are utilized to process the features. There is no dense connection in ARCN, which is for shrinking the channel numbers. There is no channel attention and there are no adaptive weights in DRDN. In ARCN, the two components are utilized to exploit the features more effectively.(2)Motivation on global and local residual learning: In ARCN, global and local residual learning strategies are jointly applied for feature exploration. The residual connections can effectively solve the gradient vanishing problem, which make the network deeper. The local residual connection in ARCB ensures the gradient flow, while the global residual learning in ARCN guarantees the identical information transmission, which improves the network capacity and representation.

4.1. Experiment
4.1.1. Implementation Details

In proposed ARCN, there are K = 16 ARCBs stacked as the main path of the nonlinear mapping module. The squeeze step in channel attention shrinks the channel number as n = 16. For all convolution layers expect for upscale block, there are c = 64 filters.

Our ARCN is trained on DIV2K [34] dataset. DIV2K dataset is a novel dataset for SISR problem. There are 1000 images with up to 2K resolution from real world. In DIV2K, there are 800 images for training, 100 images for validation, and 100 images for testing. In this paper, we train our ARCN with 800 training images and validate the model with 5 images. The paired training data are cropped with resolution of 48 48 for LR patches. The batch size for training is set as 20. The model is updated for 1000 epochs by Adam optimizer. The learning rate of Adam optimizer is set as and halved for every 200 iterations.

Five testing benchmarks are used to evaluate the network performance. They are Set5 [35], Set14 [36], B100 [37], Urban100 [38], and Manga109 [39]. The evaluation indexes are chosen as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM).

4.1.2. Experiment Results

We compare our ARCN with some SISR works: SRCNN [6] and FSRCNN [7]. The quantitative PSNR/SSIM comparisons are shown in Table 2.

From Table 2, our model has achieved competitive or better performance on five benchmarks compared to other works. For Urban100 and Manga109, our ARCN achieves better performance than the others. There are high-resolution images from real world in Urban100, and Manga109 is composed of the comic book covers. From this point of view, our ARCN could recover the complex structural textures more effectively.

Visualization comparisons are shown in Figure 5. There are four images chosen from Urban100 testing benchmark to compare the performance. From the visualization, our ARCN performs better than the others on structure texture recovery.

4.2. Model Analysis

(1)Study on parameters: From the design, our proposed flexible upscale block could save the parameters. To show the comparison on parameter and performance, we test the model on five benchmarks. The quantitative results are shown in Table 3. We compare our ARCN with several recent lightweight works for SISR problem: CARN [43], EDSR-baseline [9], SRMDNF [42], and DRCN [11]. The results show that our ARCN could achieve competitive or better performance with fewer parameters. There are around 18.7% parameters off in our ARCN with similar performance.(2)Study on adaptive factors: To demonstrate the effect of adaptive factors, we illustrate the learned features from two different parts of ARCB. As shown in Figure 6, (a) denotes the features processed from main path; (b) denotes the input feature from shortcut. The shortcut contains the origin information of input features, while the processed feature concentrates on the high-frequency information on features. After the adaptive fusion, the high-frequency information will be enhanced by aggregation.Furthermore, to demonstrate the learned weights of different ARCBs, we show in Figure 7. The x-axis denotes the depth of ARCBs. The y-axis denotes the learned weights. From the figures, with the increase of network depth, the exploited high-frequency information will be more important with larger . This demonstrates the capacity of adaptive residual learning, which learns a flexible weight for different level of information. A quantitative comparison of PSNR/SSIM with and without is shown in Table 4. From the results, acts as an important role for performance improvement.(3)Study on upscale block: In this paper, we propose an efficient upscale block for arbitrary upscaling factors. As a substitution, the vanilla cascading design and proposed efficient one hold competitive performance. However, the proposed upscale block holds a much smaller number of parameters. A comparison of two upscale blocks with different scaling factors is shown in Table 5. From the table, proposed blocks have much fewer parameters with less computation cost for upscaling the images.

5. Conclusion

In this paper, we proposed a novel adaptive residual channel attention network named ARCN for single image super-resolution (SISR) problem. In the proposed ARCN, adaptive residual channel attention block (ARCB) was designed for better performance. Mixture factors in ARCB were learned while training, which weighted the information from two paths in blocks adaptively. Channel attention mechanism was introduced to ARCB for distributing the importance among different channels. Besides ARCB, a tiny but flexible upscale block design was proposed for different scaling factors. Experimental results showed that our proposed ARCN could not only achieve competitive or better performance with fewer parameters than other lightweight works but also recover the complex structural textures more effectively.

In the future, more reference-free perceptual assessments will be performed to demonstrate the network performance. Furthermore, more experiments will be conducted on real-world datasets.

Data Availability

The image and quantitative comparison data used to support the findings of this study are included within the article.

Conflicts of Interest

The authors declare that they have no conflicts of interest.


This paper was supported by the Science Foundation of Shenyang University of Chemical Technology (LQ2020020).