Abstract
To accelerate the data acquisition speed of magnetic resonance imaging (MRI) and improve the reconstructed MR images’ quality, we propose a parallel MRI reconstruction model (SPIRiT-Net), which combines the iterative self-consistent parallel imaging reconstruction model (SPIRiT) with the cascaded complex convolutional neural networks (CCNNs). More specifically, this model adopts the SPIRiT model for reconstruction in the k-space domain and the cascaded CCNNs with dense connection for reconstruction in the image domain. Meanwhile, this model introduces the data consistency layers for better reconstruction in both the image domain and the k-space domain. The experimental results on two clinical knee datasets as well as the fastMRI brain dataset under different undersampling patterns show that the SPIRiT-Net model achieves better reconstruction performance in terms of visual effect, peak signal-to-noise ratio, and structural similarity over SPIRiT, Deepcomplex, and DONet. It will be beneficial to the diagnosis of clinical medicine.
1. Introduction
Magnetic resonance (MR) imaging (MRI) is a modern clinical medical tool with multiparameter, multisequence, multidirectional, and high soft tissue resolution. But acquiring fully sampled k-space data tends to take a long time. Reducing the number of k-space data can accelerate the acquisition speed, but undersampled k-space data will cause artifacts in the reconstructed image. Therefore, how to reconstruct MR images with high quality from undersampled k-space data has become a significant problem.
Parallel imaging is an effective way to accelerate the acquisition speed, which uses a set of multichannel coils to acquire data simultaneously and encode spatial information using the sensitivity differences of the coils to reduce the number of necessary gradient encoding steps and the time required for imaging [1]. Reconstruction methods for parallel MRI mainly include the sensitivity encoding (SENSE) [2], the generalized auto-calibrating partially parallel acquisitions (GRAPPA) [3], and the iterative self-consistent parallel imaging reconstruction (SPIRiT) [4]. Among them, SPIRiT is a widely used coil-by-coil autocalibrating model for parallel MRI, which avoids the sensitivity information estimation in the SENSE reconstruction model due to the implicit use of sensitivity information. To improve the reconstructed images’ quality, the SPIRiT model was combined with the joint total variation regularization term [5, 6], simultaneous 2D low rankness in k-space [7], nonlocal low-rank constraints [8], and joint sparsity and sparsifying transform learning [9].
Recently, deep learning-based methods have been proposed to reconstruct parallel MRI from undersampled k-space data. Deep learning models for MR image reconstruction can be categorized into two main groups: one of them is the end-to-end nonexpanding iterative methods that directly use a standard network structure to learn the mapping from input to output. Specifically, Lee et al. [10] proposed a deep residual learning network composed of magnitude and phase networks separately trained for reconstructing parallel MRI. Huang et al. [11] proposed a variant of U-net with channel-wise attention (CWA) as reconstruction block together with long skip connection technique. Sriram et al. [12] proposed an end-to-end variational network for parallel MRI reconstruction. Yiasemis et al. [13] proposed a recurrent variational network (RecurrentVarNet) for reconstructing parallel MR images via multiple recurrent blocks. In recent years, other networks such as transformer [14] and diffusion model [15] have also been used in MRI reconstruction tasks to further improve the quality of reconstruction and achieve good results.
The other is a physical model-based method that starts from a hypothetical optimization problem and unfolds the iterative optimization algorithm to obtain a deep network. Specifically, Hammernik et al. [16] proposed a variational network was proposed to learn critical parameters for parallel MRI reconstruction from fully sampled multicoil k-space data. Aggarwal et al. [17] proposed a model-based reconstruction framework with a deep learned prior (MoDL), which involved data-consistency terms and learned convolution neural networks (CNNs) to capture redundant information of images. Sriram et al. [18] proposed the GRAPPA-Net model by combining the GRAPPA model with deep neural networks to achieve improved reconstructions. Pramanik et al. [19] proposed a model-based deep learning framework (Deep-SLR) to accelerate the structured low-rank (SLR) matrix-completion algorithm significantly, which trained a CNN-based filterbank to estimate the annihilation relations.
To improve the reconstruction performance, we propose the SPIRiT-Net model that simultaneously utilizes the correlations of multicoil k-space data and complex convolutional neural networks (CCNNs) to capture redundant information of parallel MR images. Specifically, the SPIRiT calibration block and the k-space data consistency layer are used to reconstruct the undersampled k-space data, and then the cascaded CCNNs with dense connections and image domain data consistency layers are used to further reconstruct the calibrated k-space data.
Our main contributions are as follows: First, we use SPIRiT for initialization of undersampled k-space data. Second, we use a CCNN with dense connection for further capturing the redundant information of the MR images reconstructed by SPIRiT. Third, we conduct experiments on two clinical knee datasets as well as the fastMRI brain dataset. The experimental results show that our model achieves better results in terms of visual effects and quantitative metrics compared to the SPIRiT, Deepcomplex, and DONet models.
The outline of this article is given as follows. We propose a parallel MRI reconstruction model (SPIRiT-Net) by combining the cascaded CCNNs with the SPIRiT model in Section 2. Simulated experimental results and analysis are given in Section 3. Finally, Section 4 summarizes this paper.
2. Methods
2.1. Notation and Preliminaries
In this article, the domain of real numbers and the domain of complex numbers are denoted as and , respectively. The relevant mathematical manipulation operators are denoted by Eulerian letters, the matrices are denoted as boldface capital letters, and some basic constants are denoted by unbolded capital letters. We provide the symbols used in Algorithm 1 proposed in this paper along with their corresponding meanings in Table 1.
| ||||||||||||||||||||||||||||||||||||||||||||||||||
2.2. Problem Formulation
The undersampled k-space data of a multicoil image are given by the following equation:where denotes the multicoil image to be reconstructed, and denote the real and imaginary parts of , denotes the multicoil undersampled k-space data, and denote the real and imaginary parts of , denotes the total number of received coils, denotes the number of single coil undersampled data points, denotes the total number of pixel points of the image to be reconstructed, and denote the number of rows and columns of the image. denotes the system encoding matrix, and denote the undersampling operator and Fourier transform operator for multicoil images. is an identity matrix of size , denotes the Kronecker product, denotes the undersampling matrix sampled from k-space, and denotes the 2D Fourier transform operator [8] for single-coil images. The parallel MRI reconstruction based on deep learning can be represented as follows:where denotes the forward mapping function of the deep learning network parameterized by , which includes millions of adjustable network weights. is the data fidelity term.
2.3. Algorithmic Framework
In contrast to real-valued natural images, MR images are complex, and each pixel point contains magnitude and phase information. Some of the networks use the magnitude image for training [20]. Some of the networks consider the real and imaginary parts separately [19]. The former methods neglect the phase information during reconstruction, and the latter methods lose the correlation between the real and imaginary parts. The CCNN can take both magnitude and phase information into account [21]. Therefore, we choose the CCNN as the benchmark network for feature learning.
The SPIRiT-Net mainly consists of a k-space-domain reconstruction block and an image-domain reconstruction block. The k-space-domain reconstruction block consists of a SPIRiT calibration block (SCB) and a k-space data consistency (KDC) layer. The image-domain reconstruction block consists of a complex-valued convolution block (CCB) and an image-domain data consistency (IDC) layer. Figure 1 illustrates the architecture of the proposed SPIRiT-Net.
2.3.1. SPIRiT Calibration Block
In the SCB, the multigroup convolution kernels are first estimated from the auto-calibration signals (ACS), and then used to reconstruct the missing k-space data (linear interpolation), as shown in Figure 2.
Let denote the coil of the k-space data , and denote the k-space value of the coil at position , denote the k-space points in the neighborhood of position , denote a set of convolution kernels estimated from the ACS, and is the conjugate transpose of . The reconstruction of the point of the coil at position is given by the following equation:
can be obtained by solving the following optimization problem:
Equation (3) can also be expressed in matrix form as follows:where is a matrix containing a series of convolution kernel operators at appropriate positions.
2.3.2. k-Space Data Consistency Layer
After the undersampled k-space data are calibrated, k-space data consistency operations are applied, as shown in Figure 3. K-space data consistency operations can be expressed as follows:where and are the operators that select the acquired and nonacquired points from the whole k-space respectively, and are the operators that put them back in the right positions in the k-space, respectively. Therefore, denotes the zero-filled k-space data.
2.3.3. Complex-Valued Convolution Block
k-space data reconstructed by SCB and KDC are fed into the image-domain reconstruction block after inverse Fourier transformation. The input of the image-domain reconstruction block is as follows:
The image-domain reconstruction block consists of cascaded CCBs and IDC layers. Each CCB contains convolution units , where , as shown in Figure 4. Each convolution unit includes a complex convolution unit and a rectified linear unit (ReLU), except the last convolution unit only contains a complex convolution unit. The convolution kernel of each complex convolution unit is of size , and and denotes the real and imaginary parts of . The output feature map for the convolution unit of the () CCB is as follows:where , , and are the input feature maps for the convolution unit of the CCB.
In addition, the output feature maps of each previous layer are concatenated to ensure that the output features can be reused. Let denote the dense concatenation operation, then the final output feature map for the convolution unit of the CCB is formulated as follows:
The final output of the CCB is as follows:
2.3.4. Image-Domain Data Consistency Layer
Each CCB is followed by an image-domain data consistency (IDC) layer to ensure acquired k-space data unchanged. The IDC layer is shown in Figure 5. The output of the IDC layer is formulated as follows:
It should be noted that the output of the IDC layer is directly used as the input of the CCB:
Then, we obtain a parallel MRI reconstruction algorithm SPIRiT-Net based on the SPIRiT model and CCNN, which is summarized in Algorithm 1.
After the final multicoil image is reconstructed, the square root of sum of squares (SOS) method is adopted to obtain a magnitude image. SOS is defined as follows:
3. Experimental Results
3.1. Experimental Setup
The knee datasets and the fastMRI brain datasets are used to evaluate the performance of all the reconstruction models [16]. The knee datasets were obtained using a clinical 3T MRI scanner with two different sequences called “Coronal Proton Density (Coronal-PD)” and “Sagittal Proton Density (Sagittal-PD),” respectively. The scan parameters of the Coronal-PD weighted sequence are TR = 2,750 ms, TF = 4 ms, TE = 27 ms, matrix size = 320 × 288 × 15, and voxel size = 0.49 × 0.44 × 3 . The scan parameters of the Sagittal-PD weighted sequence are TR = 2,800 ms, TF = 4 ms, TE = 27 ms, matrix size = 384 × 307 × 15, and voxel size = 0.46 × 0.36 × 3 . Each sequence consists of 20 subjects. Twenty slices from No. 11−30 for each subject are selected. All the slices are padded or cropped to 320 × 320 pixels to ensure that all images have the same size. Fourteen patients are randomly selected for training, three for validation, and three for testing. The fastMRI brain datasets were obtained using a clinical 3T MRI scanner. The T1 postweighted sequences are selected for experiment, 50 patients’ data for training, eight patients for validation, and eight patients for test, eight slices for each patient are selected. All the slices are also padded or cropped to 320 × 320 pixels to ensure that all images have the same size.
All compared models are trained and tested on a desktop computer with an Intel (R) Core (TM) [email protected] GHz processor, 64 GB of RAM memory, an NVIDIA GeForce RTX 3090 GPU (24 GB of memory) and an Ubuntu 20.04 operating system (64-bit). We compare the proposed model SPIRiT-Net with one traditional model SPIRiT [4] and two deep learning models (Deepcomplex [21] and DONet [22]). Except for SPIRiT implemented in Python, all other compared models are implemented using TensorFlow 2.4. The model parameters of SPIRiT-Net are shown in Table 2. Besides, the proposed SPIRiT-Net is trained using the Adam optimization algorithm and Kaiming initialization method [23]. ACS size of 24 × 24 is used for all the undersampling patterns for the SPIRiT-based algorithms. It should be noted that the small amount of unacquired data of ACS does not affect the calibration of SPIRiT.
We employ the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) for the evaluation of reconstructed images’ quality. PSNR is defined as follows:where denotes the mean square error between the ground truth image and the reconstructed image , and denotes the maximum value of .
SSIM is defined as follows:where are constants, and denote the mean values of and , respectively, and denote the variance of and , respectively, and denotes the covariance of and . Larger PSNR and SSIM values indicate better reconstruction quality.
3.2. Evaluation of Quantitative Metrics
Tables 3 and 4 tabulate the quantitative metric results (average PSNR and SSIM values with standard deviation) for the reconstructed MR images via the compared models under the 3x and 5x accelerated 1D random undersampling (1DRU) pattern, 2D random undersampling (2DRU) pattern, 2D Poisson-disc undersampling (2DPU) pattern, and radial undersampling pattern (RADU) for the Coronal-PD and Sagittal-PD datasets, respectively. As shown in Tables 3 and 4, SPIRiT-Net consistently outperforms SPIRiT, Deepcomplex, and DONet in terms of average PSNR and SSIM values in all cases. We also provide box plots of the average difference in PSNR between SPIRiT-Net and the compared models (Deepcomplex, DONet, and SPIRiT) on the Coronal-PD and Sagittal-PD datasets from the 3x and 5x accelerated 1DRU, 2DRU, 2DPU, and RADU patterns respectively, as shown in Figure 6.
(a)
(b)
(c)
(d)
Table 5 tabulates the quantitative metric results (average PSNR and SSIM values with standard deviation) for the reconstructed MR images via the compared models under the 4x and 8x accelerated 1D equispaced undersampling (1DEU) pattern for the real k-space raw data in the fastMRI brain dataset. It can be seen from Table 5 that the SPIRiT-Net model achieves some improvement in the metrics results compared to SPIRiT, Deepcomplex, and DONet models.
3.3. Evaluation of the Visual Effect
We visually compare the reconstructed MR images via all the models. The reconstructed image and its corresponding fully sampled image are utilized to plot an error map, while 2x magnification is performed at the same location on both the reconstructed image and the error map to observe the reconstruction details. We choose different undersampling patterns, acceleration factors, and datasets to observe the reconstruction results.
First, it can be seen from Figure 7 that under the 1D undersampling pattern, there are many artifacts in the reconstructed image of SPIRiT model, Deepcomplex, and DONet greatly reduce those artifacts, and our proposed SPIRiT-Net further removes the artifacts. Second, as can be seen from Figures 8−10, under the 2DRU, 2DPU, and RADU patterns, the four models achieve relatively clear reconstructed images. The Deepcomplex and DONet models achieve good results in the overall reconstruction quality. However, we can see from the zoomed-in error maps that the SPIRiT-Net model achieves the best reconstruction quality in terms of details. Third, Figure 11 shows the reconstruction results of the real k-space raw data from the fastMRI brain dataset. We can see that there are still obvious artifacts in the reconstructed image of SPIRiT. Deepcomplex and DONet models remove most of the artifacts, and the SPIRiT-Net model further removes the artifacts. The above experimental results prove that the SPIRiT-Net model can effectively remove the artifacts and improve the reconstruction MRI quality.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
3.4. Ablation Studies
To prove the effectiveness of the SCB and dense concatenation, we compare the baseline models SPIRiT and Deepcomplex, Deepcomplex with SCB, and Deepcomplex with SCB and dense concatenation (namely our proposed SPIRiT-Net) on the Coronal-PD dataset with the 2DPU pattern. It can be observed from Table 6 that introducing SCB to the baseline model Deepcomplex effectively improves the PSNR and SSIM values of the reconstructed images, while the feature reusability of the dense concatenation leads to further PSNR and SSIM improvement. This indicates that both SCB and dense concatenation are indispensable and important parts of our proposed SPIRiT-Net.
We also incorporate the SCB and dense concatenation with DONet. As shown in Table 7, DONet with SCB and dense concatenation achieves higher PSNR than DONet with SCB and the baseline models SPIRiT and DONet.
3.5. Significance Test
We perform the Wilcoxon signed-rank test in PSNR between SPIRiT-Net and the competing methods (SPIRiT, Deepcomplex, and DONet). Table 8 shows the -values of the Wilcoxon signed-rank test, and it can be seen that the -values between the proposed SPIRiT-Net and the competing methods are less than a significance level of 0.05 under all undersampling patterns and with different acceleration factors, which proves that the difference between our method and the other three competing methods is significant at a significance level of 0.05.
4. Conclusions
In this paper, we propose a parallel MRI reconstruction model (SPIRiT-Net), which combines the cascaded CCNNs with the SPIRiT model. This model simultaneously utilizes the correlations of multicoil k-space data and a CCNN to capture redundant information of parallel MRI. Simulation experiments on two clinical knee datasets as well as the fastMRI brain dataset show that SPRiT-Net outperforms SPIRiT, Deepcomplex, and DONet in terms of visual effects and quantitative metrics. Additional ablation experiments demonstrate that introducing the SPIRiT model to the deep learning network can effectively improve the reconstruction quality.
Data Availability
The knee data used to support the findings of this study have been deposited in the Zenodo repository at https://doi.org/10.5281/zenodo.10668971. The dataset contains 11−30th slices of 20 patients. Three patients (No. 7, 9, and 6) are randomly selected for validation, three patients (No. 1, 17, and 14) for testing, and other 14 patients for training. We also provide permanent identifiers (at https://doi.org/10.5281/zenodo.10556450) for our reconstruction results of the Coronal-PD knee dataset and the fastMRI brain dataset. We provide theses .h5 files of our test results for one to verify the accuracy of our metrics. We also provide the Python code for reading .h5 files and calculating PSNR metrics.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The authors would like to acknowledge Kerstin Hammernik (Coronal-PD dataset and Sagittal-PD dataset), Michael Lustig (SPIRiT code), Shanshan Wang (Deepcomplex code), and Chun-Mei Feng (DONet code) for making their in vivo datasets and codes publicly available. This work was supported in part by the Yunnan Fundamental Research Project (202301AT070452) and the National Natural Science Foundation of China (61861023).