Microscopic Image Analysis in HistopathologyView this Special Issue
A Heart Segmentation Algorithm Based on Dynamic Ultrasound
The heart is one of the most important organs of the human body. The role of the heart is to promote blood flow and provide sufficient blood flow to organs and tissues. The research on the heart has important theoretical and clinical significance. Because of the noninvasive and intuitive display of ultrasound image, it can dynamically obtain the heart state and has become the main means to detect the heart dynamics. We analyze the characteristics of cardiac ultrasound image from the medical point of view and signal processing. The heart movement is periodic and rhythmic. The image signal can be decomposed. Firstly, the image is decomposed into high- and low-frequency signals to highlight different dimensional information. Then, the attention model was introduced, focusing on the heart region. Finally, the multidimensional network carrying model was established to achieve cardiac segmentation. The experimental results show that the AOM of the algorithm proposed in this paper reaches 92%, which has a certain degree of advancement and can assist doctors to make accurate diagnosis.
The heart is an important organ of the human body. Its main function is to provide pressure for blood flow and run the blood to all parts of the body. Its efficiency is related to the shape and size of the heart . Therefore, it is very important to extract the region of the heart and carry out research. Tang et al.  proposed a clustering algorithm for heart sound segmentation. Icardo  established a model from morphology to analyze the heart. Ahn et al.  focused on the region of coronary artery through CT and carried out research. Mythili et al.  used SVM to predict heart disease. Pedrosa et al.  built a model to analyze the voice of young children. Methaila et al.  used the big data technology to analyze early heart disease. Pace et al.  implemented the whole heart segmentation interactively. Saquib et al.  diagnosed heart disease by calculating the volume ratio. Xiong et al.  combined CT images with clinical data for analysis. Wolterink et al.  used dilated convolutional neural networks to segment MR images. Arabasadi et al.  used a hybrid neural network genetic algorithm to predict heart disease. Tong et al.  constructed 3D deep supervised U-Net to segment all hearts. Ahmed et al.  built a deep network to segment the heart. Gao and Lu  focused on fetal baseline to realize classification and extraction. Xu et al.  combined a deep learning network and graph matching to realize whole heart segmentation. de Albuquerque et al.  proposed fast heart fat segmentation based on CT data set. Naseer et al.  constructed fuzzy sets to diagnose heart diseases. Yoshida et al.  used U-Net to automatically segment the heart based on CT data. Banerjee et al.  reconstructed the heart in 3D from 2D data. Diniz et al.  built Concat-U-Net to realize automatic heart extraction. Diniz et al.  built U-Net++ to realize heart segmentation. Liu et al.  proposed automatic segmentation algorithm using attentional convolutional network. Chen et al.  constructed 3D filter to suppress ultrasonic image noise. Song et al.  proposed deep networks for heart segmentation and explained the significance and challenges of heart segmentation.
Through the above introduction, the current analysis of the heart is focused on mature images with high imaging quality such as CT  and MR . However, CT and MR mostly present static images, which cannot meet the situation of dynamic analysis. Therefore, medical staff often use dynamic echocardiography in analyzing  and has achieved good results. Echocardiography is the most widely used cardiac examination method in clinic, which can dynamically evaluate the structure and function of the heart. The ventricular wall motion of the heart is an important driving force to maintain the function of the heart pump. It is very important for doctors to identify whether the ventricular wall motion is normal based on ultrasonic images. The current ventricular wall motion evaluation is mainly by eye or manual trace for motion amplitude, which is very dependent on the operator’s experience and time-consuming. If the segmental recognition method of ventricular wall can be established to quickly identify the abnormal motion of myocardium, it would be very helpful for the diagnosis of heart disease.
However, due to the large noise interference of ultrasonic image and the lower imaging quality than CT and MR, it is difficult to put forward computer-aided segmentation. In general, main problems are as follows: (1) low image quality leads to limited access to information. (2) The heart only occupies a limited area in the image, which is inefficient to process the image pixels in a unified scale. (3) The constructed cardiac ultrasound image feature network has limited carrying capacity and insufficient representation.
In view of the above shortcomings, we propose a new segmentation algorithm based on dynamic echocardiography: (1) Octconv is proposed from the perspective of signal composition, and the signal is decomposed into high- and low-frequency parts to extract different frequency information; (2) to simulate the doctor’s diagnosis process and build an attention mechanism to focus on the region of the heart; and (3) from the perspective of deep learning, a multidimensional information carrying network is constructed to fuse low-frequency and high-frequency signals to achieve heart segmentation.
The research of echocardiography algorithm is based on the current situation of low echocardiography quality and the actual situation of echocardiography clinical diagnosis. A dynamic echocardiography segmentation algorithm based on deep learning is constructed, and the specific block diagram is shown in Figure 1. Firstly, Octconv is proposed to decompose the image into low-frequency and high-frequency signals by simulating the principle of visual perception. Then, an attention mechanism is built to focus on the region where the heart is located. Finally, a multidimensional information carrying network is built to realize heart segmentation.
2.1. Image Preprocessing
Evaluating the motion of ventricular wall is an important content of echocardiography. The left ventricle is the main chamber responsible for pumping blood. Usually, the left ventricular wall is divided into 16 or 17 segments (including the apex) according American Society of Echocardiography (ASE) guidelines. Each segment was dominated by its corresponding coronary artery branches for blood supply. Segmental wall motion abnormalities can occur in coronary heart disease or other myocardial diseases (diabetes, hypertension, amyloidosis, etc.); thus, each LV segment is needed to be evaluated. This cardiac segmentation can help doctors quickly identify the region of ventricular wall with abnormal motion, so as to accurately locate the location of impaired myocardium.
We used video sequences to do our research. Therefore, the detection region is extracted from the pixel variation of the whole video sequence, and the sector region can be preliminarily determined. To this end, we build the maximum density projection mechanism to obtain the information of the sequence image and focus on the image region, as shown in Figure 2.
The projection of maximum density is the maximum value of the coordinates at () and can be used to calculate the area of concern.
After the maximum density projection of the image, the threshold method was used to determine detection area, and then, the ultrasonic image region was obtained according to mathematical morphology operation.
Information can be transmitted at different frequencies, and the characteristic diagram output by the convolution layer can also be regarded as the combination of information at different frequencies. Therefore, scholars have proposed Octconv to store and process feature graphs with low spatial resolution and slow spatial change. Octconv has orthogonality and complementarity and can establish a better topology to reduce the spatial redundancy caused by low-frequency information in deep convolution network. The specific structure is shown in Figure 3. is the proportion of input low-frequency channel in the total channels. The corresponding input size is , and the output size is . is the convolution kernel, and the corresponding parameter is , , and . The most available calculation amount is . Compared with ordinary convolution, Octconv contains a certain proportion of low-frequency channels, and the complexity is improved. But the performance is better than ordinary convolution.
Octconv uses the coefficient to append the feature graph into high-resolution component () and low-resolution component (). saves the edge, contour, and other details of the images with a large amount of information data. saves the abstract information of the images with a small amount of information data. The multifrequency feature representation method is constructed, and the smooth low-frequency mapping is stored in the low-frequency tensor to reduce spatial redundancy. The specified convolution kernel is used to convolute to obtain the corresponding component:
High-frequency output is , and low-frequency output is . The amount of Octconv convolution calculation is only 1/4 of that of ordinary conv.
2.3. CBAM Attention Module
Due to the increase of the length of neural network, the long-distance information will be weakened, resulting in the loss of important information after information transmission. To reduce the risk of losing important information, we focus our limited attention on important information to build attention modules. Due to the convolutional block attention module (CBAM) combining channel and space, the channel attention module and spatial attention module are connected in series. According to the importance of features, they focus on the area of interest step by step and extract key information through corresponding spatial transformation. The structure is shown in Figure 4. (1)Channel attention module. Input feature and two channel features () are obtained through Maxpool and Avgpool to connect MLP (multilayer perceptron). The corresponding number of neurons in the first layer is , and the number of neurons in the second layer is . The weight coefficient of the activation function is obtained as follows:
The corresponding new feature is (2)Spatial attention module. Maxpool and Avgpool are performed on to get the channel description, and the dimension is . Splice the two descriptions according to the channel, and pass through convolution, and the weight function is
The corresponding new feature is
where and form a complementary relationship. The use of Maxpool and Avgpool increases more diversified information, resulting in less computation and stable performance improvement of CBAM.
2.4. Network Model
The main advantage of U-Net++ is that it connects each branch U-Net to share a coding layer and allows the middle part of the model to participate in training, so that the information loss in the encoder process can be repaired to some extent. The performance is improved on the basis of limited increase of parameters. Since each branch U-Net shares an encoder path, the same information is lost in the downsampling process. U-Net++ obtains U-Net features with different depths in the decoder process, but this feature difference is obtained after each branch U-Net performs its own upsampling, so the information supplement of U-Net++ in the feature recovery process is limited and has lack of pertinence.
Through the above analysis, there are five layers in total based on the U-Net++ framework. Octconv is used to replace the traditional Conv2D. The whole framework is divided into two U-Net++ structure branches of high-frequency and low-frequency synchronous parallel. Octconv is used in each layer of coding-decoding to exchange low-frequency and high-frequency information. It reduces the model parameters, reduces the information loss introduced by each coding layer in downsampling, and makes the whole network obtain more abundant information. In the decoding stage, bilinear interpolation sampling is used to realize image restoration. After each sampling, it is spliced with the features of the same layer and the same scale and then connected with CBAM to strengthen the attention of convolution operation in the model to the target area and realize more accurate pixel category classification. The problem of offset of restoration features is avoided as much as possible in order to make the network more robust.
Init-Octconv and input high frequency obtain characteristics through downsampling, and then, Octconv is performed to obtain high-frequency and low-frequency output. COM-Octconv is input by high-frequency and low-frequency characteristics, and Octconv is directly carried out through hyperparameter to control the proportion of high- and low-frequency channels of output. Fin-Octconv is input by high-frequency and low-frequency characteristics. After convolution, the low-frequency upsampling is added to the high frequency to output the high frequency.
We introduce the objective function. Tversky coefficient is the generalized coefficient of Dice coefficient and Jaccard coefficient:
where represents the true value and represents the predicted value. In medical image segmentation, Dice Loss () is often used in small lesion segmentation as the objective function. It has good performance in the case of extremely unbalanced samples, but in general, its use will have an adverse impact on back propagation and make the training unstable. As a result, the effect of Dice Loss is not ideal and the fluctuation range is large.
Therefore, we construct a new objective function to reduce the weight of simple samples and increase the weight of difficult samples. coefficient is introduced to learn difficult samples with small sample regions of interest:
3. Experiment and Result Analysis
The experiment is conducted with 50 groups of image sequences collected by the hospital, and the data resolution is , and the equipment is Philips Epiq 7c. As shown in Figure 5, the red box is the continuous image sequence. The area where the heart is located is marked at the pixel level by a professional doctor. We adopt double-blind method, which is a common algorithm in annotation field. The gold standard was determined by combining the labeling results of two physicians. The programming environment is Linux and Python. We adopted the deep learning network. So the input of the deep learning network we designed is . For other resolutions, normalize to . We built 1 : 1 ratio of training samples to test samples. The cross-validation method is adopted.
3.1. Octconv Performance
Based on the principle of signal processing, we decompose signals into low-frequency and high-frequency components to construct Octconv. To verify Octconv performance, we build a unified network model and compare the convergence curves of Octconv and ordinary convolution to obtain the performance. AOM is introduced to measure the convergence effect.
As shown in Figure 6, Octconv has better iteration times than traditional convolution when the curve converges. Due to the orthogonality and complementarity of Octconv, signals are decomposed into low-frequency and high-frequency components, which is in line with the principle of visual sensing and has low redundancy. However, the traditional convolution only considers the relationship between parts, resulting in a large amount of redundancy in the parameters and slow convergence speed.
3.2. Segmentation Effect Comparison
The loss function, crossentropy loss function, Dice Loss, and Tversky Loss are proposed as the objective function for experiments. We introduce AVM, AUM, and CM to measure the accuracy of the algorithm:
whereis the segmentation result marked by the doctor andis the segmentation result of the algorithm, in which AOM and CM are proportional to the segmentation result and AVM and AUM are inversely proportional to the segmentation result. The performance is shown in Table 1. Crossentropy loss is used in multitarget segmentation and analyzed from the perspective of energy, but the signal-to-noise ratio of ultrasonic image is low and the energy is not concentrated, resulting in poor performance. Dice Loss is often used in medical small target segmentation loss function, but the image area occupied by heart data is large, which has the risk of overfitting, resulting in inaccurate boundary focusing. Tversky comprehensively considers the differences between simple samples and complex samples and constructs weights, which has better improved the case of fuzzy boundary and reduced the probability of pixel misclassification.
In order to verify the performance of different network algorithms, Sen and Spe are introduced to measure:
where TP represents the number of pixels correctly predicted as positive samples and TN represents the number of pixels correctly predicted as negative samples. FP represents the number of pixels incorrectly predicted as positive samples, and FN represents the number of pixels incorrectly predicted as negative samples. The results are shown in Table 2, and the corresponding ROC curve is shown in Figure 7.
To visually display the segmentation effect, as shown in Figure 8, the pretreatment results are shown in green, and the obtained sector area is the detection area.
The heart is periodic diastolic and contractile, which can be determined according to the correlation of motion. It is on this basis that the significance region is determined to achieve auxiliary segmentation. Traditional U-Net can inevitably lose details after downsampling. Encoder-decoder structure can reduce the loss, but after the image restoration by upper sampling, it is difficult to pay attention to the details and determine the contour limits of the target category under the influence of multinoise. The proposed attention model pays more attention to the target region, highlights the difference between foreground and background, and realizes accurate segmentation.
In this paper, the deep learning network is adopted, and the results are related to the input image. Therefore, if the acquisition equipment is changed, the image quality will be different and the algorithm should be retrained, but the overall framework remains unchanged, so the algorithm should have certain effects.
The heart is the human body’s important organ and has very important sense to the dynamic monitoring. Based on dynamic echocardiographic noninvasive, can display the superiority, is currently the main observation way of the heart. However, it is limited by the mechanism of ultrasonic imaging; imaging noise is large and easy to produce motion blur. We propose a new heart segmentation algorithm, which decomposes the image into low-frequency and high-frequency signals according to the principle of signal decomposition to obtain different frequency information. According to the principle of cardiac dynamic contraction, the attention model was introduced to simulate the diagnosis process of doctors and focus on the region of the heart. The network model will be built to realize multiscale information carrying and finally realize heart segmentation. Subsequent studies on myocardial extraction and analysis will be carried out to study the lightweight network, improve the calculation efficiency of the algorithm, and quickly assist doctors to make accurate diagnosis.
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
This work is supported by Key project of Shaanxi International Science and Technology Cooperation Foundation (2022kwz-19) and Xi’an Science and Technology Project (21YXYJ0105).
J. T. Des Jardin, J. Chikwe, R. T. Hahn, J. W. Hung, and F. N. Delling, “Sex differences and similarities in valvular heart disease,” Circulation Research, vol. 130, no. 4, pp. 455–473, 2022.View at: Publisher Site | Google Scholar
H. Tang, T. Li, T. Qiu, and Y. Park, “Segmentation of heart sounds based on dynamic clustering,” Biomedical Signal Processing and Control, vol. 7, no. 5, pp. 509–516, 2012.View at: Publisher Site | Google Scholar
J. M. Icardo, “The teleost heart: a morphological approach,” Ontogeny and Phylogeny of the Vertebrate Heart, pp. 35–53, 2012.View at: Publisher Site | Google Scholar
S. J. Ahn, D. K. Kang, J. S. Sun, and M. H. Yoon, “Accuracy and predictive value of coronary computed tomography angiography for the detection of obstructive coronary heart disease in patients with an Agatston calcium score above 400,” Journal of Computer Assisted Tomography, vol. 37, no. 3, pp. 387–394, 2013.View at: Publisher Site | Google Scholar
T. Mythili, D. Mukherji, N. Padalia, and A. Naidu, “A heart disease prediction model using SVM-decision trees-logistic regression (SDL),” International Journal of Computer Applications, vol. 68, no. 16, 2013.View at: Google Scholar
J. Pedrosa, A. Castro, and T. T. Vinhoza, “Automatic heart sound segmentation and murmur detection in pediatric phonocardiograms,” in 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2294–2297, Chicago, IL, USA, August 2014.View at: Publisher Site | Google Scholar
A. Methaila, P. Kansal, H. Arya, and P. Kumar, “Early heart disease prediction using data mining techniques,” Computer Science & Information Technology Journal, vol. 24, pp. 53–59, 2014.View at: Google Scholar
D. F. Pace, A. V. Dalca, T. Geva, A. J. Powell, M. H. Moghari, and P. Golland, “Interactive whole-heart segmentation in congenital heart disease,” International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, vol. 9351, pp. 80–88, 2015.View at: Publisher Site | Google Scholar
N. Saquib, M. T. I. Papon, I. Ahmad, and A. Rahman, “Measurement of heart rate using photoplethysmography,” in 2015 International Conference on Networking Systems and Security (NSysS), pp. 1–6, Dhaka, Bangladesh, January 2015.View at: Publisher Site | Google Scholar
G. Xiong, P. Sun, H. Zhou et al., “Comprehensive modeling and visualization of cardiac anatomy and physiology from CT imaging and computer simulations,” IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 2, pp. 1014–1028, 2017.View at: Publisher Site | Google Scholar
J. M. Wolterink, T. Leiner, M. A. Viergever, and I. Išgum, “Dilated convolutional neural networks for cardiovascular MR segmentation in congenital heart disease,” Reconstruction, Segmentation, and Analysis of Medical Images, Springer, Cham, pp. 95–102, 2016.View at: Google Scholar
Z. Arabasadi, R. Alizadehsani, M. Roshanzamir, H. Moosaei, and A. A. Yarifard, “Computer aided decision making for heart disease detection using hybrid neural network-genetic algorithm,” Computer Methods and Programs in Biomedicine, vol. 141, pp. 19–26, 2017.View at: Publisher Site | Google Scholar
Q. Tong, M. Ning, W. Si, X. Liao, and J. Qin, “3D deeply-supervised U-Net based whole heart segmentation,” International Workshop on Statistical Atlases and Computational Models of the Heart, Springer, Cham, pp. 224–232, 2018.View at: Google Scholar
R. S. Ahmed, J. Liu, Z. Fei, and M. Zahid, “Automated segmentation of whole cardiac CT images based on deep learning,” International Journal of Advanced Computer Science and Applications, vol. 9, no. 4, pp. 1–8, 2018.View at: Publisher Site | Google Scholar
W. Gao and Y. Lu, “Fetal heart baseline extraction and classification based on deep learning,” in 2019 International Conference on Information Technology and Computer Application (ITCA), pp. 211–216, Guangzhou, China, December 2019.View at: Publisher Site | Google Scholar
X. Xu, T. Wang, Y. Shi et al., “Whole heart and great vessel segmentation in congenital heart disease using deep neural networks and graph matching,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 477–485, Cham, October 2019.View at: Publisher Site | Google Scholar
V. H. C. de Albuquerque, D. D. A. Rodrigues, R. F. Ivo et al., “Fast fully automatic heart fat segmentation in computed tomography datasets,” Computerized Medical Imaging and Graphics, vol. 80, p. 101674, 2020.View at: Publisher Site | Google Scholar
I. Naseer, B. S. Khan, S. Saqib, S. N. Tahir, S. Tariq, and M. S. Akhter, “Diagnosis heart disease using Mamdani fuzzy inference expert system,” EAI Endorsed Transactions on Scalable Information Systems, vol. 7, no. 26, 2020.View at: Google Scholar
A. Yoshida, Y. Lee, N. Yoshimura, T. Kuramoto, A. Hasegawa, and T. Kanazawa, “Automated heart segmentation using U-Net in pediatric cardiac CT,” Measurement: Sensors, vol. 18, p. 100127, 2021.View at: Google Scholar
A. Banerjee, J. Camps, E. Zacur et al., “A completely automated pipeline for 3D reconstruction of human heart from 2D cine magnetic resonance slices,” Philosophical Transactions of the Royal Society A, vol. 379, no. 2212, p. 20200257, 2021.View at: Publisher Site | Google Scholar
J. O. B. Diniz, J. L. Ferreira, O. A. C. Cortes, A. C. Silva, and A. C. de Paiva, “An automatic approach for heart segmentation in CT scans through image processing techniques and Concat-U-Net,” Expert Systems with Applications, vol. 196, p. 116632, 2022.View at: Publisher Site | Google Scholar
J. O. B. Diniz, D. A. Dias Júnior, L. B. da Cruz et al., “Heart segmentation in planning CT using 2.5 D U-Net++ with attention gate,” Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, pp. 1–9, 2022.View at: Publisher Site | Google Scholar
H. Liu, W. Chu, and H. Wang, “Automatic segmentation algorithm of ultrasound heart image based on convolutional neural network and image saliency,” IEEE Access, vol. 8, pp. 104445–104457, 2020.View at: Publisher Site | Google Scholar
H. Chen, H. Xu, P. Shi et al., “3-D Gabor-based anisotropic diffusion for speckle noise suppression in dynamic ultrasound images,” Physical and Engineering Sciences in Medicine, vol. 44, no. 1, pp. 207–219, 2021.View at: Publisher Site | Google Scholar
Y. Song, S. Ren, Y. Lu, X. Fu, and K. K. Wong, “Deep learning-based automatic segmentation of images in cardiac radiography: a promising challenge,” Computer Methods and Programs in Biomedicine, vol. 220, p. 106821, 2022.View at: Publisher Site | Google Scholar
M. M. Rahaman, C. Li, Y. Yao et al., “Identification of COVID-19 samples from chest X-ray images using deep learning: a comparison of transfer learning approaches,” Journal of X-Ray Science and Technology, vol. 28, no. 5, pp. 821–839, 2020.View at: Publisher Site | Google Scholar
N. Oyama-Manabe, T. Aikawa, S. Tsuneta, and O. Manabe, “Clinical applications of 4D flow MR imaging in aortic valvular and congenital heart disease,” Magnetic Resonance in Medical Sciences, vol. 21, no. 2, pp. 319–326, 2022.View at: Publisher Site | Google Scholar
M. Lisi, M. Cameli, G. E. Mandoli et al., “Detection of myocardial fibrosis by speckle-tracking echocardiography: from prediction to clinical applications,” Heart Failure Reviews, pp. 1–11, 2022.View at: Publisher Site | Google Scholar