Abstract
Retinal image mosaic is the key to detect common diseases, and the existing image mosaic methods are difficult to solve the problems of low contrast of fundus images and geometric distortion between images in different fields of view. To solve the problem of noise in retinal fundus images, an image mosaic algorithm based on the genetic algorithm was proposed. Firstly, a series of morphological pretreatment was performed on the fundus images. Then, the vascular network is extracted by obtaining the maximum entropy of the image to determine the threshold value. The similarity of the image to be spliced is a feature, and the genetic algorithm is used to solve the optimal parameters to achieve the maximum similarity. By smoothing the image, a clear image with minimum noise is obtained. Experimental results show that the proposed algorithm can effectively realize the image mosaic of the fundus. The method proposed in this paper can provide support for high-precision automatic stitching of multiple single-mode color fundus images.
1. Introduction
Retinal fundus images are often used as an important basis for judging fundus diseases and systemic diseases. This is because fundus diseases and systemic diseases can cause certain fundus changes and appear in different retinal blood vessels on the retina of the fundus [1, 2]. However, due to limitations of fundus photography techniques, the fundus images obtained tend to have small angles and narrow fields of view. Clinical doctors often compare and analyze multiple fundus images. It is very necessary to multiple image mosaic to help doctors obtain more accurate and complete fundus image information [3–6].
The synthesized large image is more complete and has higher resolution than the small image. Since different small image acquisition devices are different and the scenes are different, there are problems in various aspects such as the division ratio, the color difference, and the angle. These problems lead to significant differences in the human structure presented in the image. Therefore, the image must be properly processed to achieve the purpose of image mosaic.
For image mosaic, domestic and foreign scholars have done a lot of research, and a variety of image registration methods based on various features are proposed. Theoretical support for the efficient image mosaic is provided [7].
Representative methods proposed by scholars include the automatic splicing method based on vessel branching and crossover features [8, 9]. An algorithm is proposed that utilizes features such as retinal branches and intersections and then performs automatic mosaic and target tracking [10]. Most of these traditional methods adopt the method based on feature correspondence. The image registration is completed by extracting feature points. However, the registration effect is closely related to the pretreatment effect, so there are still some problems in the practical application [11, 12].
In this paper, a new method of automatic fundus image mosaic is proposed, which is based on a large number of studies at home and abroad. This method firstly carries on the morphological treatment to the image of the fundus and sets the threshold value to extract the retinal vessels. Then, the genetic algorithm is used to obtain the registration parameters on the binary segmentation images of blood vessels. Finally, image registration and image mosaic are realized.
2. Related Work
2.1. Fundus Image Feature Point Detection and Registration
Feature point matching is a key step in image registration, and its matching accuracy determines the final registration result. At present, the classical feature detection methods are SIFT algorithm and SURF algorithm [13–15]. SIFT feature is an image local feature description operator based on scale space proposed by Lowe [16], which remains invariant to image scaling, rotation, and affine transformation. The basic idea is to detect feature points by establishing Gaussian difference scale space, determine the location and scale of feature points, then use the main direction of the neighborhood gradient of the feature point as the direction feature, and generate a 128-dimensional feature descriptor as the matching basis. The SIFT method is used to purify feature points; feature descriptors with scale invariance, rotation invariance, and affine invariance are generated for each feature point; the feature descriptor can match the feature points of adjacent images with high quality. The superior performance of the SIFT feature extraction algorithm determines that it will become the best choice. Like other methods, it will not be limited by the influence of different shooting equipment or viewing angle change. Image distortion, translation, and scaling can be well solved for it. The high dimension of the descriptor is a disadvantage of SIFT in the matching step. For real-time applications on ordinary PC, the three stages of detection, description, and matching are faster.
The SURF algorithm is similar to the SIFT method, but its feature descriptor has only 64 dimensions and is much faster. The SURF algorithm is an accelerated and improved version of the SIFT algorithm. The Hessian matrix using the SURF algorithm can greatly improve the matching speed. At the same time, any matching algorithm can improve the speed by reducing the dimension of descriptors. The steps of the SURF algorithm are mainly divided into constructing Hessian matrix, constructing scale space, feature point location, feature point main direction allocation, and generating descriptor.
How to select the feature extraction algorithm has its own criteria. The first point is that the number of feature points should not be too large, the distribution should be uniform, and the distribution of feature points should not be too dense; otherwise, too many feature points will slow down the calculation speed. The second point is that you cannot select only one point as the feature point; otherwise, it will lead to large deviation. The third is that shooting an image will lead to the change of angle and the distance of image viewing angle. The feature points we select should not change with these deformation or scaling; that is, stability is required.
Generally, the objects we get are small images obtained from different perspectives or shooting backgrounds, and they are all about the same object or scene [17, 18]. The operation of transforming these images to the same coordinate system and then matching is called image registration. This operation needs to calculate the function data required for image transformation, and it can also be used to find similar parts in two images. In the process of image registration, first the part of the object to be registered is determined, the overlapping area is framed, a feature point extraction method is selected using the abovementioned method, the feature points are selected and calculated, and then they are used for registration. The accuracy and registration effect of image registration technology directly affect the information accuracy and visual effect of the final image. At present, the feature-based image registration method is widely used, but this method only uses a small part of image gray information. In case of errors in the process of feature extraction and feature matching, the final result will also be misguided. Therefore, the results of this method largely depend on the extraction accuracy of feature points and the accuracy of feature point matching.
2.2. Fundus Image Mosaic Method
After multiple images are changed to the same base map by the fixed method, the similar parts are combined, the size and deformation are adjusted, and the image is merged into one image according to the relative position. The image contains all the above information, and this operation is called image fusion [19, 20]. Image fusion is used to process the transition region in image mosaic, which mainly solves the following two problems: firstly, the brightness or color of several images to be processed is different, which will produce uneven phenomenon of the fused image and greatly affect the final result. Secondly, due to the geometric aberration caused by the change of viewing angle and shooting with different equipment, ghosting will occur during fusion.
The commonly used image fusion methods are direct average method, weighted average method, median filter method, multiresolution analysis method, and so on. The improved weighted average method is adopted in this paper. After converting multiple images to the same coordinate system, the registration between images is completed, and their transformation matrix is obtained [21, 22]. The improved weighted average algorithm is used to fuse different retinal images, and then the median filtering method is used to eliminate the discontinuity of light intensity so that the stitching pattern looks more natural. In this way, the part with abnormal brightness on the edge of the overlapping area is solved.
The basic process of image mosaic is shown in Figure 1.
3. Image Preprocessing
3.1. Smooth Filer
The edge and other high frequency components of the image can be filtered by smooth filter without changing the low frequency components so that the gray changes of the image become smooth. Due to the linear characteristics of blood vessels [23, 24], the image is smoothed by using linear structural elements whose length is close to the maximum diameter of the blood vessels.
According to the size of vascular width in the image to be spliced, structural elements of size are selected.
The image is divided into 12 directions at an angle of 15°. The original image is separately processed in each direction by structural elements, and the maximum value of the results in twelve directions is solved. The maximum value is denoted by the new image grayscale, and the obtained smooth image is . The formula is as follows:where is a structural element, is the original image, is the computed image, is the smoothed image, is the neighborhood radius, and is the number of operations, and its value can be determined experimentally.
3.2. Image Enhancement
The smoothed filtered image has a smoother grayscale. The contrast between the vascular features and the background is low, which is not conducive to subsequent registration and splicing. Therefore, the processed image needs to be further enhanced.
The vascular image conforms to the morphological features, and the Top-Hat operation in morphology is an effective high-pass filter [25]. By filtering out areas of the image where the grayscale changes relatively smoothly, selecting the vascular elements in the retinal image allows the blood vessels to be extracted from the entire image. The Gaussian filter operator smoothes the extracted vascular elements, and the Laplacian enhances the edge of the blood vessels. These two filter operators are classically effective operators in image enhancement. Therefore, these three operators are used for image enhancement in the image enhancement process of this paper.
In order to make the contrast between the blood vessel and the background in the image larger, this paper accumulates the corresponding results in 12 directions after performing Top-Hat operation on the image. After the enhancement of the method, the retinal vascular network is substantially separated from the background, and the vascular network can be obtained by directly dividing the image by threshold.
3.3. Threshold Segmentation
In this paper, the concept of entropy in information theory is applied to image segmentation. The fundamental purpose is to select an appropriate threshold to ensure the successful separation of blood vessels from the background and to ensure that the information volume of both the blood vessel image and the background image is maximum.
If the gray level of the image to be spliced is M, the posterior probability is expressed as follows:where is the total number of pixels in the image and is the number of pixels whose gray level is . Let the threshold be , and set the gray level of the image as two groups, respectively, as and .
The entropy of the whole image is expressed as follows:where and is the threshold, which makes reach the maximum value. This threshold is used to segment the image.
4. Image Registration and Mosaicing
4.1. Feature Value Selection
For the registration of the fundus image, an alignment method based on feature correspondence is adopted. And the blood vessel information of the retina is extracted as a feature for registration. In this paper, the vascular network characteristics in the image will be used as the registration feature.
The essence of image registration is to find a spatial transformation so that the similarity of the two images is maximized after the transformation. According to the form of transformation, general registration transformation models include affine transformation, nonlinear transformation, and projection transformation. In the registration process, different transformation models are selected, and the mapping relationship obtained is also different. In order to overcome the problem of image scale deformation caused by image rotation or movement, the bilinear transformation is used in this paper. The definition is shown as follows:where are the transformation parameters. Preprocessed by the images mentioned above, binary images and can be obtained. The registration of the target images and is translated into registration of the images and . The introduction of the parameter represents the similarity measure of the two images. Therefore, the registration of the image is expressed as follows:where is a spatial transformation. After determining the spatial transformation type and similarity measure, the registration process is to seek an optimal value to maximize the image similarity measure. The optimal value can be expressed as follows:
4.2. Genetic Algorithm for Obtaining Registration Parameters
In order to ensure the maximum image overlap area, the genetic algorithm is chosen to solve the problem. The genetic algorithm is a global optimization algorithm, which can effectively avoid the problem of falling into local optimal solution. It is an efficient and parallel search method for solving problems. To find image registration parameters, it usually requires solving the problems of establishment of fitness function. Under the premise of binary value, the corresponding correlation value is valid only when the corresponding pixel is a natural number. Therefore, for the image registration of the retina, the objective function defined by equation (6) is the fitness function of this paper.
Choosing the appropriate operator can accumulate good performance in the population and obtain more perfect individuals. It can be ensured that the excellent individuals have a greater chance to pass on the good performance to their offspring. In this paper, the method of single-point crossing is adopted. The steps can be expressed as selected intersection⟶cross operation⟶interchange operation⟶effect proofreading. The mutation operator is used to generate new individuals whose operation is to randomly determine the experimental individual. According to the above principles, determine the orientation of the mutation and then carry out correlation coding and value processing.
4.3. Image Mosaicing
The image registration method was determined in the previous section and achieved good results. Image registration solves the problem of positional differences between different images, but it does not change the difference in brightness or chromaticity between images. Because the brightness of different images is different or the edge distortion is caused by the collection environment, the image is not suitable for direct splicing. In order to avoid the obvious mosaic marks caused by the image’s own shooting factors, the overlapping areas of the image need to be smoothed. In this paper, the following formula is used to smooth the image:where is the width of the overlap region when the leftmost end of the image overlap region is the coordinate zero point.
5. Experiments and Results
In this paper, a gene population with the chromosome number of 50 is selected. The code length of a single individual is 54. The probability of crossover and mutation is 1.0 and 0.01, respectively, and the number of iterations is N = 100. When realizing multiple image registration, first take the fundus image in front of each group of images as the reference image, register with any other image according to the above steps, take the result image as the reference image, and then register with the third image until the final result is obtained. In the registration process, there are no specific requirements for the splicing sequence of other images except that the front image is taken as the reference image during the first splicing.
In order to quantitatively analyze the registration accuracy of fundus images, this paper uses the root mean square error (RMSE) and mean absolute error (MAE) between all feature point pairs that are accurately matched and transformed as the evaluation indexes [26, 27], which are defined as follows:where is the matching point of the reference image, is the corresponding matching point of the target image after coordinate transformation, and is the logarithm of matching points.
In order to illustrate the effectiveness of this method, ten groups of color fundus images are tested by the methods of literatures [14] and [17, 21], respectively. The and between matched feature point pairs are calculated and compared with this method. The comparison results are shown in Table 1.
Figure 2 is obtained by matching the images based on the above parameters. Figures 2(a) and 2(b) are images obtained by dividing the registration image to be performed separately. Figure 2(c) is the image obtained by mapping Figures 2(a) and 2(b) after obtaining the first generation of optimal parameters and superimposing them. As shown in the figure, the superposition of the two images is very obvious, and the image background is very messy. Figure 2(d) is an image obtained after final iteration and optimization processing. Comparing Figures 2(c) and 2(d), it can be found that the background of the image becomes clear and the messy texture is cleared, indicating that the positional difference of the image is eliminated. Therefore, the parameters obtained by this method have a very obvious registration effect on the image, and the image is ideally matched.
(a)
(b)
(c)
(d)
After registration, this paper uses the proposed fusion algorithm to fuse all registration results. Figure 3 shows several groups of fusion results. As shown, Figures 3(a) and 3(b) are two matched retinal images. Figure 3(c) is an effect diagram obtained by directly splicing images. Figure 3(d) is a mosaic effect diagram after smoothing. Comparing with Figure 3(d), Figure 3(c) has prominent marks in the splicing area with obvious shadows. After smoothing, there is no obvious trace of the image overlap area. It is indicated that the smoothing treatment has a good effect on eliminating the mosaic traces of the image.
(a)
(b)
(c)
(d)
In order to further illustrate the effectiveness of this method as a whole, this method is compared with the methods in literatures [14] and [17, 21], and the correct registration logarithm RMSE of any two images in the same group is used as the evaluation criteria.
It can be seen from Table 2 that the method in this paper has the best registration result and strong registration ability and can be applied to multiple image stitching.
6. Conclusion
Aiming at the problems of time-consuming and large amount of calculation in the existing retinal image mosaic technology, a fundus image mosaic method based on the genetic algorithm is proposed in this paper. The fundus image mosaic algorithm proposed in this paper performs a series of preprocessing on the image, and it combines the maximum entropy theory to extract the retinal network to obtain a binary image. In the image registration stage, the common vascular bifurcation points and intersections are discarded. The similarity of vascular network is used as the registration feature, and the optimal parameters of registration are obtained by the genetic algorithm. The experimental results showed that the proposed algorithm can effectively achieve the splicing of fundus images and eliminate the seams generated in the splicing to some extent. Aiming at the problem of low contrast of fundus image, the algorithm adjusts the threshold of the algorithm and improves the purification process after feature point matching. It not only overcomes the shortcomings of other methods but also realizes the fast and high-precision automatic stitching of fundus images. In the fundus image fusion algorithm, we need to further find and explore a suitable improvement method to improve the fusion effect, reduce the calculation fusion time, and improve the fusion efficiency [28, 29].
Data Availability
The labeled dataset used to support the findings of this study is available from the corresponding author upon request.
Conflicts of Interest
The author declares that there are no conflicts of interest.
Acknowledgments
This work was supported by the CERNET Innovation Project (No. NGII20190121) and in part by the Shaanxi Province Soft Science Research Plan Project (No. 2021 krm117).