Abstract

In this paper, we propose a rapid rigid registration method for the fusion visualization of intraoperative 2D X-ray angiogram (XA) and preoperative 3D computed tomography angiography (CTA) images. First, we perform the cardiac cycle alignment of a patient’s 2D XA and 3D CTA images obtained from a different apparatus. Subsequently, we perform the initial registration through alignment of the registration space and optimal boundary box. Finally, the two images are registered where the distance between two vascular structures is minimized by using the local distance map, selective distance measure, and optimization of transformation function. To improve the accuracy and robustness of the registration process, the normalized importance value based on the anatomical information of the coronary arteries is utilized. The experimental results showed fast, robust, and accurate registration using 10 cases, each of the left coronary artery (LCA) and right coronary artery (RCA). Our method can be used as a computer-aided technology for percutaneous coronary intervention (PCI). Our method can be applied to the study of other types of vessels.

1. Introduction

Recently, cases of coronary artery disease have significantly increased owing to the extension of the average life expectancy and lack of exercise [1, 2]. Coronary artery disease is caused by the narrowing or closing of the coronary artery due to the stenosis and metabolic failure of a heart muscle [2]. A stent enthesis is used as a representative treatment method. In a stent enthesis, a metal epiploon is inserted and unfolded into a diseased artery to relieve stenosis [2, 3]. A stent enthesis is a noninvasive treatment method, which has advantages for a patient such as the minimization of physical and mental strain through a minimal incision, control, and anesthesia. [4]. However, as this treatment depends on 2D X-ray angiogram (XA) images and 3D vascular structures—which are understood primarily by medical doctors through intuition and haptic feedback—the accuracy of this difficult medical procedure is not guaranteed [5]. To compensate for these weaknesses, an assistance method to register and visualize both preoperative 3D computed tomography angiography (CTA) and 2D XA images in real time has been actively researched.

Extensive research has been performed on the 2D and 3D registration of coronary arteries. Kerrien et al. measured translational parameters between 2D and 3D images using normalized cross correlation (NCC) [6, 7]. Distortion coefficient, rotational angle, and scaling parameters were calculated using a calibration procedure. However, the method required extensive processing time for the exploration of the whole image during the optimization of NCC. Additionally, the 2D and 3D images were acquired from the same equipment. Hipwell et al. proposed a similarity measurement method using pattern intensity (PI), gradient correlation, and gradient difference for 2D and 3D registration with a best neighbor optimization algorithm [8]. Although this method was accurate, it required extensive processing time. Metz et al. generated digitally reconstructed radiograph (DRR) images from CTA images and registered DRR and XA images using the NCC similarity measure [9, 10]. To minimize the deformation difference of vessels due to the heartbeat, 4D CTA images taken in multiphase were exploited. However, 4D CTA images are not usually taken in the general procedure of an intervention because of the high exposure to radiation compared to 3D CTA images. Additionally, the initial positions of vessels were manually aligned. Again, the NCC optimization procedure using a multi-resolution gradient ascent optimization technique required extensive processing time. Benseghir et al. performed 2D and 3D registration by comparing the similarities between a curved segment which was divided at branching or terminal points [11]. This method used Frechet distance and iterative closest curve (ICC). This method considered the geometric similarity between curves of vessels instead of only points—as in ICP. However, the computational efficiency was significantly degraded when considering multiple curves. Furthermore, a local minimum may have been encountered as this method did not consider the global search. Kaila et al. proposed a Gaussian mixture model of 3D centerlines from biplane XA and CTA images and maximized a posterior probability for registration using coherence point drift (CPD) [12] and expectation-maximization (EM) [13]. The robustness of this method was increased by using vessel diameter and branch point information. However, it required extensive processing time for 3D space registration and could not be applied to the general, single XA image. Kim et al. extracted vascular centerlines from XA and CTA images and projected 3D vessels onto 2D vessels for registration [14, 15]. In this method, the distances between the projected 3D vessels and 2D vessels were minimized by iteratively performing rotation, translation, and scaling transformations. The accuracy was further improved by performing thin plate spline-robust point matching (TPS-RPM) [16], which considers the local deformation of vessels. However, in this method, 3D vascular deformation was not considered because registration was performed only after the centerlines of 3D CTA vessels were projected onto the 2D images. Therefore, the topology information of the 3D CTA vessels was not preserved during registration. Additionally, owing to the lack of depth information, the convergence of local minima caused registration errors. Park et al. [17] proposed an initial registration using the principal axis generation and alignment with the bounding box such as a chessboard. For the accurate and fast fine registration, the trilateration method and Powell’s optimization were performed. However, there is a disadvantage that the alignment of the cardiac cycles between two images is not taken into account and should be adjusted manually. Additionally, the principal axis used in the initial registration and the trilateration method caused registration errors due to the difference of the 2D and 3D segmentation, noise, etc. in the process of generation.

To address these limitations, in this paper, we propose a fast and accurate registration method for intraoperative 2D XA and preoperative CTA images of the same patient. Our method consists of three steps. First, to minimize the differences between two images, alignment of the cardiac cycle is performed based on electrocardiography (ECG) information. Second, a simulation environment is built similar to the acquisition timing based on DICOM (digital imaging and communications in medicine) information, and the gross transformation mismatch is corrected by optimal cube registration, including vessels. As this initial registration aligns the global rotation, scaling, and translation parameters between two images, it does not require distance map generation for the entire image. As such, the region of the distance map that uses local distance propagation is minimized. Following initial registration, faster and more robust convergence to the optimal value is achieved as the search space is limited to near the optimal value. The subsequent fine registration employs the selective distance measurement as a similarity measure to find the optimal transformation parameters and to minimize the distance between vessels. During fine registration, the importance value according to the anatomical structure of vessels is defined and exploited, resulting in more robust registration which is not affected by the difference of 2D and 3D segmentation, noise, etc. The proposed method minimizes unnecessary operations and enables fast and accurate registration by optimizing the performance of each step. We use 20 clinical datasets to evaluate the performance of our method in comparison to the registration accuracy, speed, and robustness achieved by Kim et al. [15] and Park et al. [17].

The remainder of this paper is organized as follows. Section 2 describes the proposed registration method of 2D and 3D images—this procedure consists of three processing steps. Section 3 presents the experimental results, followed by concluding remarks and a discussion of future work in Section 4.

2. Registration Method between 3D and 2D Images

The proposed method consists of the following steps, which are illustrated in Figure 1. For the registration of 3D CTA and 2D XA images of the same patient, each obtained from a different apparatus, cardiac cycle alignment and spatial alignment should first be performed. The ECG information of the 3D CTA and 2D XA images is used for this purpose. Subsequently, using DICOM information, we align the registration space by applying the environment while the 2D XA image is acquired into the 3D CTA image. However, as displacement information is missing from the DICOM information, an optimal boundary box is used to compensate for the lack. In order to perform fine registration, local distance propagation is performed to generate a 2D local distance map, and registration between the 3D CTA and 2D XA images is performed through an optimization process that uses selective distance measurement. In our method, the overall registration process uses a 3D centerline extracted from the 3D CTA image [18, 19] and 2D centerline extracted from the 2D XA image [20]. We input the normalized importance value into the 3D centerline and use it for registration. This feature reduces the probability of convergence to the local minimum, which allows for more robust convergence.

2.1. Matching of the Cardiac Cycle Using the ECG Information

The preoperative 3D CTA image may be acquired at any time during the cardiac cycle. Conversely, the intraoperative 2D XA images are acquired continuously. As such, a difference in vascular shape can occur between the images due to the different acquisition times in the registration process. Therefore, as shown in Figure 2, the process of matching the image acquisition times should first be performed.

In order to match the CTA and XA image acquisition times, an R-R interval is extracted from the ECG curve of the XA image and frames corresponding to the interval are distinguished. Subsequently, the cardiac cycle acquisition time of the CTA image is matched to the most similar frame of XA image. If it is matched with multiple frames, that by which the error is most minimized through comparison of the 3D and 2D centerlines is selected.

2.2. Setting of the Importance Value according to Anatomical Structure of the Vessel

In this study, to improve the accuracy and robustness of registration, we input the normalized importance value based on the anatomical information of the coronary arteries into the 3D centerline extracted from the CTA image. We reflect the standardized anatomical information of coronary arteries, as recommended by the Society of Cardiovascular Computed Tomography (SCCT) [21], to ensure a general applicability. The importance value is applied by dividing the coronary arteries into main branch and sub-branch. The main branch is defined as RCA, LAD, and LCX based on the skeletonization information of Han et al. [18, 19]. Each main branch is divided into three areas of proximal, mid, and distal based on SCCT coronary segmentation information. Unlike the main branch, a sub-branch is not clearly distinguishable. Based on the main branch and SCCT coronary segmentation information, it is distinguished whether an unclear sub-branch exists mutually for each of the proximal, mid, and distal regions. If the number of sub-branches in each area is small or the same, it is considered to be an active sub-branch; otherwise, it is regarded as an inactive sub-branch. The importance value of the main branch is assigned to [1, 0.1] from the proximal to distal, as shown in Figure 3. The active sub-branch is gradually decreased from the bifurcation point of the main branch. The inactive sub-branch is assigned a value of 0.1.

The importance value of the 3D centerline enables the division of the main and sub-branches of the vessel. This feature reduces registration errors that converge to the local minimum because of low importance or unnecessary sub-branches. As such, it is possible to achieve robust registration.

2.3. Initial Registration between 2D and 3D Images

For multimodality image registration, alignment of the registration spaces should first be performed. For this purpose, we apply DICOM information obtained from the C-arm (the imaging apparatus of the 2D XA image) to the extracted centerline of the 3D CTA image. Subsequently, it is possible to quickly and accurately align the registration spaces of the 2D XA and 3D CTA images, using equation (1), for the initial registration:where are the coordinates of the 2D image, are the coordinates of the 3D image, is the focal length, and are the pixels/mm, which are used when converting physical space into pixel space, and denote the principal points, and and denote the variables used for rotation and translation, respectively. Table 1 shows the DICOM parameter information used in the initial registration.

However, because the DICOM information obtained from the C-arm lacks translation variable information, it is necessary to acquire an estimated value. For this reason, we calculate the minimum and maximum values of the and axes of the projected 3D and 2D centerlines; then, we create a boundary box with (, ) and (, ) as the vertices, respectively. In addition, in order to consider a heartbeat and the geometric transformation in the registration process, we generate an optimal boundary box with added margin , as shown in Figure 4. Subsequently, an initial estimate value of the translation variable is obtained by matching the optimal boundary boxes of the two images.

2.4. Generation of 2D Distance Map by Local Distance Propagation

In our method, as the global displacement between 2D and 3D centerlines is aligned during initial registration, a distance map for the entire area is unnecessary. To generate a 2D distance map, we consider eight-neighbor relations for distance propagation, as shown in equation (2). Let be the propagated distance value of the pixel at the -th position through the Euclidean distance, and the minimum distance value is allocated through comparison with the existing/conventional distance value. A distance is assigned to the pixels whose value is not propagated:

Figure 5 shows a 2D local distance map. The line displayed in white indicates the centerline extracted from the 2D XA image. The pixels having the same distance value from the centerline are displayed in the same color.

It is possible to measure one-step distance transformation, and it becomes unnecessary to calculate the distance of pixels with a distance value . Therefore, the processing time required to generate the distance map is minimized.

2.5. Similarity Measurement through Selective Distance Measurement

To identify similarities between the 3D and 2D centerlines, we use the distance value that is commonly used in the evaluation function of feature-based registration. The evaluation function has a minimum value when the 3D and 2D centerlines are aligned. For measurement of the evaluation function, we use the average of absolute distance difference (AADD), as shown in the following equation:where is the total number of 3D centerlines projected onto the 2D XA image, denotes the -th position of the 3D centerline projected onto the 2D XA image, is the importance value of , and denotes the 2D local distance map. In our method, the 3D centerline is superimposed upon the 2D local distance map [22], enabling the map to be utilized for the distance measurement between the two centerlines. As such, the processing time required to measure similarities between the centerlines is minimized.

2.6. Optimization of Transformation Function

Even if the same environment is applied at the XA image acquisition time using the DICOM information, initial registration errors can occur because of differences in protocols between imaging devices or other external factors. In order to reduce these errors, fine registration is subsequently performed using the transformation function of Powell’s optimization method [23].

The transformation function consists of a translation vector (, , and ) on the , , and axis directions and a central rotation vector (, , and ) on the , , and axes, respectively. Optimization of transformation vectors simultaneously is inefficient with respect to both processing time and accuracy. In order to optimize the efficient transformation function, the translation-only and rigid transformation are sequentially performed, as shown in the following equation:where and . In this study, once the initial registration is performed, the rotation and translation vectors in the transformation function are restricted as shown in equation (5). Additionally, we apply Powell’s optimization method [23] to converge to the optimal position. As such, it is possible to enable fast and accurate registration:where and represent the threshold of rotation and translation vectors, respectively. These values are experimentally determined.

3. Experimental Results

In this study, we implement the proposed algorithm in C++ on a PC using Visual Studio 2010, installed on a Windows7 64bit operating system. The experiments are performed with an Intel® Core™ i7 3.4 GHz CPU and 8 GB of main memory. We test our method on 10 XA LCA and 10 XA RCA images from 20 patients. The XA and CTA data are obtained using a Philips digital C-arm and Siemens SOMATOM definition flash. The image size of each XA and CTA image is 512 × 512 pixels. The acquisition rate of the XA image is 15 fps. Tables 2 and 3 contain specific information on the remaining 2D XA and 3D CTA images.

3.1. Evaluation of Registration Accuracy

The parameters for the rotation and translation vector threshold and of the transformation function are and 13 mm, respectively. The parameter is obtained by semiautomatic registration using the additional 2D-3D dataset, which is twice the mean of the measurements. Figure 6 shows the result of the process applying the proposed method of the LCA of case 9 and RCA of case 18.

Figure 7 shows the results of the proposed method for cases 1 through 20. In Figure 7, the first and second rows show the LCA results, whereas the third and fourth rows show the RCA results. The centerline color denotes depth information: shallow and deep vessel positions are indicated by blue and red, respectively, as shown in Figure 8. It is possible to visually confirm that the vessels of the preoperative 3D CTA and intraoperative 2D XA images are well-aligned, as shown in Figure 7.

To evaluate the accuracy of our algorithm, we compare the registration results of our method with the ground truth—manually annotated by the experts. The error of average difference is found to be minimal, even if it is aligned with an incorrect vessel. In this study, we use additional markers to measure the robustness of registration. The markers are manually input by the 3 experts on the 3D centerline and ground truth for the same corresponding point, respectively. The robustness of registration is measured using the average of distance difference (ADD) between markers, as shown in the following equation:where and represent the markers of the 3D centerline and the ground truth, represents the Euclidean distance between and , and represents the number of markers. 10 pairs are used for each patient.

Table 4 shows the accuracy and robustness results of the proposed method. It can be seen that performance related to the RCA is lower than that of LCA. This is because the variability of the RCA exceeds that of the LCA. As such, even if the same patient’s data are used, the registration error of the RCA will be higher than that of the LCA.

3.2. Comparison with the Previous Method

In this study, we compare the registration accuracy and speed of our method with those of the methods suggested by Kim et al. [15] and Park et al. [17]. Experiments are performed on the same environment and data. For the accuracy evaluation, the ADD between the registration result and the ground truth is measured for the centerline, marker, and bifurcation points. The centerline accuracy results for the proposed method, Kim et al. [15], and Park et al. [17] are 1.2520 mm, 1.2237 mm, and 1.4098 mm, respectively. These values are observed to be numerically similar. However, as shown in Figure 9(a), the bifurcation points do not align in Kim et al. [15], where some vessels converge to a local minimum in a nonrigid registration process [15]. Additionally, the registration method proposed by Park et al. [17] shows the incorrect registration result due to the local error in the initial and correction registration process using the principal axis as shown in Figure 9(b).

Figure 10 shows the accuracy and robustness results of the proposed method and previous methods. The proposed method is observed to be generally more accurate than the previous methods. In particular, accurately depicted vessel bifurcations are important for the precise insertion of surgical instruments and to understand the structure of the vessel during PCI. Therefore, if the accuracy is low, the vessel structure becomes difficult to analyze and understand.

Figure 11 shows the average execution times for registration using the proposed and previous methods. The average execution times of registration for the proposed method, Kim et al. [15], and Park et al. [17] are 0.2970 s, 1.7063 s, and 0.6712 s, respectively, in 20 clinical datasets. This demonstrates that the execution speed of the proposed method is faster than that of the previous methods. In this study, we perform fast and robust initial registration through the effective utilization of DICOM information. Additionally, we reduce the distance map generation time through local distance propagation. In the fine registration process, as initial registration has been performed, the search space for registration is restricted. As such, it is possible to further reduce the computation time. The proposed method minimizes unnecessary operations and enables fast and accurate registration by optimizing the performance of each step.

The registration method proposed by Kim et al. [15] attempts to improve accuracy through deformation of part of the vessel. However, it converges to the local minimum at the wrong place in a region with a local error, resulting in incorrect registration results. In particular, because a 3D centerline is projected once onto a 2D image and registration is performed only in 2D, the topology information of the 3D centerline extracted from a 3D CTA image cannot be preserved. In the registration method proposed by Park et al. [17], the overall registration process is influenced by the principal axis, thereby causing incorrect registration results for some dataset. Alternatively, in this study, we input the normalized importance value into the 3D centerline and use it for registration. This feature reduces the probability of convergence to the local minimum, which allows for more robust convergence. Furthermore, we can perform more accurate and robust registration between the 3D and 2D centerlines by preserving the topology information of the 3D centerline.

Table 5 shows the comparison of the previous methods with our method. The table shows that our number of test subjects exceed the other papers presented lately. Accuracy assessment metrics vary from paper to paper, and the target annotations (e.g., manual segmentation of vessels) also have a large variation so that it is very hard to find global criteria for comparison. However, we have tried to measure the accuracy, robustness, and clinical applicability of the registration by measuring the centerline, marker, and bifurcation of the vessel, respectively. As a result of the measurement, it can be confirmed that our method performed accurate, robust, and fast vessel registration with large test subjects. As such, our method can be used as a computer-aided technology for PCI.

4. Conclusion

To compensate for the limited 3D structure and depth information in 2D XA images, in this paper, we proposed a fast and accurate registration method for the fusion visualization of intraoperative 2D XA and preoperative 3D CTA images of the same patient. The proposed method consisted of three steps, namely, cardiac cycle alignment, initial registration, and fine registration.

First, to minimize the differences between two images according to the cardiac cycle, the cardiac cycles of the two images were aligned based on ECG information. Subsequently, a simulation environment was built similar to the acquisition timing, and the gross transformation mismatch was corrected by optimal cube registration, including vessels. An acquisition environment of 2D XA images in C-arm equipment was applied to 3D CTA images using DICOM information. As such, it was possible to quickly and accurately align the registration environment. An optimal boundary box was generated and aligned for the 3D and 2D vascular centerlines by rapidly compensating translational mismatch. As the initial registration aligned the global rotation, scaling, and translation parameters between the two images, this method lead subsequent registration to faster and more robust convergence to the optimal value. Additionally, the method did not require distance map generation for the whole image. As such, the region of the distance map using local distance propagation was minimized. During fine registration, the importance value according to the anatomical structure of vessels was defined and exploited, resulting in more robustness. The proposed method minimized unnecessary operations and enabled fast and accurate registration by optimizing the performance of each step. The experimental results showed registration errors of the whole vessel, anatomical landmarks, and branching points to be 1.252 mm, 1.458 mm, and 1.709 mm, respectively. The average processing time was 0.297 s. The proposed method performed fast and accurate registration between 2D XA and 3D CTA images, demonstrating the potential to provide doctors with substantial assistance during cardiac intervention. Future work will focus on a 2D + t/3D registration method by considering the correlation between 2D + t XA images.

Data Availability

They are clinical datasets, which cannot be opened publicly.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This research was partly supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (No. 2017R1A2B3011475). And, this work was partly supported by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea Government (MSIT) (No. 2017-0-018715, Development of AR-based Surgery Toolkit and Applications). And, this work was partly supported by the Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean Government (MSIT) (2016-0-00009, Authoring Platform Technology for Next-Generation Plenoptic Contents).