Cardiology Research and Practice

Cardiology Research and Practice / 2020 / Article
!A Corrigendum for this article has been published. To view the article details, please click the ‘Corrigendum’ tab above.

Research Article | Open Access

Volume 2020 |Article ID 2469281 |

Peng Wang, Hongliang Cong, Ying Zhang, Yujie Liu, "Comparison of the CAMI-NSTEMI and GRACE Risk Model for Predicting In-Hospital Mortality in Chinese Non-ST-Segment Elevation Myocardial Infarction Patients", Cardiology Research and Practice, vol. 2020, Article ID 2469281, 6 pages, 2020.

Comparison of the CAMI-NSTEMI and GRACE Risk Model for Predicting In-Hospital Mortality in Chinese Non-ST-Segment Elevation Myocardial Infarction Patients

Academic Editor: Robert Chen
Received24 Apr 2020
Revised13 Jun 2020
Accepted18 Jun 2020
Published24 Jul 2020


Introduction. The ability of risk models to predict in-hospital mortality and the influence on downstream therapeutic strategy has not been fully investigated in Chinese Non-ST-segment elevation myocardial infarction (NSTEMI) patients. Thus, we sought to validate and compare the performance of the Global Registry of Acute Coronary Events risk model (GRM) and China Acute Myocardial Infarction risk model (CRM) and investigate impacts of the two models on the selection of downstream therapeutic strategies among these patients. Methods. We identified 2587 consecutive patients with NSTEMI. The primary endpoint was in-hospital death. For each patient, the predicted mortality was calculated according to GRM and CRM, respectively. The area under the receiver operating characteristic curve (AUC), Hosmer–Lemeshow (H–L) test, and net reclassification improvement (NRI) were used to assess the performance of models. Results. In-hospital death occurred in 4.89% (126/2587) patients. Compared to GRM, CRM demonstrated a larger AUC (0.809 versus 0.752, ), less discrepancy between observed and predicted mortality (H–L χ2: 22.71 for GRM, and 10.25 for CRM, ), and positive NRI (0.3311, ), resulting in a significant change of downstream therapeutic strategy. Conclusion. In Chinese NSTEMI patients, the CRM provided a more accurate estimation for in-hospital mortality, and application of the CRM instead of the GRM changes the downstream therapeutic strategy remarkably.

1. Introduction

Non-ST-segment elevation myocardial infarction (NSTEMI) is a leading cause of mortality, morbidity, and hospitalization from cardiovascular disease both worldwide and in China, which has a major influence on health economies [14]. Unfortunately, lower rather than higher risk NSTEMI patients are more likely to receive more aggressive therapeutic strategies, which is the so-called risk-treatment paradox [57]. To diminish the impact of this paradox, the current guidelines consider the risk assessment by the Global Registry of Acute Coronary Events (GRACE) risk model as a fundamental component to select the most appropriate therapeutic strategy for NSTEMI patients [13]. However, several observations have suggested that the performance of the GRACE risk model (GRM) was unsatisfactory among Chinese NSTEMI patients [8, 9]. Recently, a novel risk model has been developed for the risk evaluation of NSTEMI patients [8] based on multicenter data from the China Acute Myocardial Infarction (CAMI) registry [10]. To date, the CAMI-NSTEMI risk model (CRM) has not been systematically validated in an external cohort. Moreover, it has not been fully demonstrated how these risk assessment models influenced clinical management. Thus, the present study tends to validate and compare the performance of GRM and CRM and investigate the impacts of the two proposed models on the selection of downstream therapeutic strategies in Chinese NSTEMI patients.

2. Materials and Methods

2.1. Study Participants

The details of inclusion and exclusion criteria for the study participants are illustrated in Figure 1. According to the most recent guidelines, NSTEMI was defined as symptoms of ischemia and detection of an elevation of cardiac troponin values without new persistent ST-segment elevation [13]. Three thousand and twenty-five patients with a primary clinical diagnosis of NSTEMI were included. Then, the patients were excluded if they fulfilled one of the following criteria: data missing (217), acute or chronic infectious diseases (116), cancer (68), and acute cerebrovascular disease (37). Among the 217 patients excluded from the analysis for missing data, medical histories for 68% (147/217), results of blood test for 28% (61/217), and clinical presentations for 24% (52/217) were not available. At last, 2587 patients were included in this study and were divided into in-hospital survival and death groups. This observational study complied with the Declaration of Helsinki and was approved by the local Ethics Committees.

2.2. Data Collection and Definitions

The following variables were registered through review of the electronic medical record: age, sex, history of diabetes mellitus, hypertension, hyperlipidemia, smoking, previous revascularization, and previous myocardial infarction. The body mass index (BMI), Killip class, heart rate (HR), systolic blood pressure (SBP), creatinine (Cr), white blood cell count (WBC), cardiac arrest, and ST-segment depression on an 18-lead electrocardiogram at admission were also recorded. Hyperlipidemia was defined as a total cholesterol of at least 220 mg/dl, low-density lipoprotein cholesterol of at least 140 mg/dl, fasting triglycerides of at least 150 mm/dl, or receiving treatment with oral lipid-lowering agents. Diabetes was defined as fasting glucose levels over 7 mmol/l or treatment currently with diet, oral glucose-lowering agents, or insulin. The smoking status included current smoker, nonsmoker, and previous smoker (quit >6 month). The primary endpoint of this study was in-hospital death, defined as all-cause death during hospitalization.

2.3. Risk Assessment by the CRM and GRM

The GRM included 8 independent risk factors: age, Killip class, SBP, ST-segment depression, cardiac arrest, Cr, initial cardiac enzyme findings, and HR [11]. The initial cardiac enzyme findings were positive because all patients have been diagnosed with NSTEMI. According to the GRM and guideline recommendations, we classified patients into 3 risk groups: the low-, medium-, and high-risk group [1]. The CRM identified 11 independent predictors of in-hospital mortality: age, BMI, SBP, Killip class, cardiac arrest, ST-segment depression, Cr, WBC smoking status, previous MI, and previous percutaneous coronary intervention, and all patients were classified into 3 risk groups based on the CRM [8].

2.4. Statistical Analysis

All statistical analyses were carried out by MedCalc (version 15.2.2; MedCalc Software, Mariakerke, Belgium) and R (version 3.2.4; R Foundation for Statistical Computing, Vienna, Austria). A two-tailed value less than 0.05 was considered statistically significant. Continuous variables were compared using Student’s t-tests or Mann–Whitney U-tests as appropriate. Count variables were assessed using the χ2-test or Fisher’s exact test as appropriate. This study conducting validation and comparison of multivariable prediction models strictly followed Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement [12]. To validate and compare the predictive value of the GRM and CRM, we used three characteristics: discrimination, calibration, and classification [13]. Discrimination refers to how well the model differentiates those having an endpoint from those not having. The area under receiver-operator characteristic curve (AUC) was used to quantify the improvement in discrimination [14]. Calibration reflects the extent to which the values predicted by the model agree with the observed values. We used Hosmer–Lemeshow (H–L) tests which divided patients into ten groups according to deciles of mortality and calculated a chi-square statistic (H–L χ2) to assess calibration [15]. As the selection of downstream therapeutic strategy usually bases on the risk classification, we established a reclassification table to evaluate the net reclassification improvement (NRI), determining how correctly a model reclassifies patients into various risk categories compared with another [16].

3. Results

Baseline characteristics of the study cohort are listed in Table 1. Overall, hospital death occurred in 4.89% (126/2587) patients. Compared to the in-hospital survival group, the in-hospital dead group were older, had more male, lower BMI and SBP, and a higher prevalence of diabetes mellitus, hypertension, current smoker, cardiac arrest, high Killip class, and ST-segment depression. Also, HR, Cr, WBC, and troponin T were higher in the in-hospital dead group. All the differences were statistically significant ().

CharacteristicTotal (n = 2587)In-hospital dead (n = 126)In-hospital survival (n = 2461) value

Age61.91 ± 10.9772.84 ± 10.6961.35 ± 9.74<0.0001
Male1290 (50)84 (67)1206 (49)0.0001
BMI (kg/m2)21.24 ± 3.9121.14 ± 3.2323.25 ± 3.74<0.0001
Diabetes mellitus540 (21)48 (38)492 (20)<0.0001
Hypertension1462 (57)84 (67)1378 (56)0.0194
Hyperlipidemia587 (23)21 (17)566 (23)0.1146
Previous revascularization
 PCI185 (7)3 (3)182 (7)0.0628
 CABG45 (1.7)1 (1)44 (1.8)0.5846
Previous MI325 (12)30 (24)295 (12)0.0027
Smoking status<0.0001
 Current smoker855 (33)18 (14)837 (34)
 Previous smoker393 (15)24 (19)369 (15)
 Nonsmoker1339 (52)84 (67)1255 (51)
HR (beats/min)78.94 ± 29.4790.01 ± 27.2378.37 ± 21.14<0.0001
SBP (mmHg)127.70 ± 34.93109.24 ± 33.73128.65 ± 22.19<0.0001
Cr (μmol/L)85.67 ± 98.64144.64 ± 93.1782.65 ± 63.38<0.0001
WBC (109/L)8.33 ± 8.0716.45 ± 7.667.91 ± 4.25<0.0001
Troponin T (ug/l)0.26 ± 2.241.47 ± 4.280.20 ± 1.37<0.0001
Cardiac arrest58 (2)9 (7)49 (2)0.0007
Killip class<0.0001
 I1598 (62)23 (18)1575 (64)
 II463 (18)20 (16)443 (18)
 III329 (13)34 (27)295 (12)
 IV197 (7)49 (39)148 (6)
ST-segment depression1315 (51)84 (67)1231 (50)0.0003

Values are presented as mean ± SD or n (%). BMI: body mass index; CABG: coronary artery bypass grafting; PCI: percutaneous coronary intervention; MI: myocardial infarction; HR: heart rate; SBP: systolic blood pressure; Cr: creatinine; WBC: white blood cell count.

The receiver-operator characteristic curves of 2 models are exhibited in Figure 2. The AUC for the CRM (0.809, 95% confidence interval: 0.789 to 0.829, ) was significantly () larger than that for the GRM (0.752, 95% confidence interval: 0.729 to 0.774, ).

The predicted mortality was compared with the observed mortality in deciles of predicted mortality as illustrated in Figure 3. The GRM manifested a predominance of underestimation, resulting in a poor calibration (H–L χ2 = 22.71, ). The differences between the observed and predicted mortality were not evident in H–L calibration plots for CRM, so the calibration of the CRM was good (H–L χ2 = 10.25, ).

Table 2 shows the reclassification table comparing the CRM to GRM. For the 126 positive patients, compared to the GRM, the CRM correctly reclassified 23 from the medium- to high-risk category, 12 from low to high, and 6 from low to medium, but 2 from medium to low and 2 from high to medium. Of the 2461 negative patients, 174 were correctly reclassified to a lower risk category but 82 to a higher PTP category. As a result, compared to the GRM, the NRI for the CRM was 0.2937 in positive, 0.0374 in negative, and 0.3311 overall (). In other words, the replacement of the GRM by CRM for every 2587/(23 + 12 + 6 + 174)≈13 patients would result in 1 correct reclassification.

Risk category based on the CRMTotalReclassificationNRI

Risk category based on the GRM
Negative patients0.03330.07070.3311<0.0001

Positive patients0.32540.0317

The classification of patients by the CRM was compared to that by the GRM. NRI = (P (Up|Positive)-P (Down|Positive)]-[ P (Up|Negative)-P (Down|Negative)). A positive patient was defined as a patient who died in hospital. NRI: net reclassification improvement; GRM: GRACE risk model; CRM: CAMI-NSTEMI risk model.

4. Discussion

This observational analysis determined that the CRM provided a more effective prediction for in-hospital mortality in Chinese NESTEMI patients. Compared to the GRM, the CRM demonstrated a larger AUC, less discrepancy between the observed and predicted mortality, and a positive NRI. It was worth noting that the application of the CRM instead of the GRM may pronouncedly change the downstream therapeutic strategy in these NESTEMI patients.

Compared to ST-segment elevation myocardial infarction (STEMI) patients, NSTEMI patients present with more heterogeneous variation in ischemic risk and comorbidities so that the risk classification has been considered as a fundamental component to select the most appropriate therapeutic strategy for NSTEMI patients [13, 17]. However, in clinical practice the treatment-risk paradox is widespread [57], which is partly due to the suboptimal risk assessment underestimating the ischemic risk [18, 19], especially in China [4, 6, 20]. In conformity with this, according to the H–L calibration plots in the present study, the GRM dramatically underestimated the in-hospital mortality and classified nearly 40% in-hospital death into the low- or medium-risk category, which may cause underuse of invasive strategy for patients at high risk [1, 2]. Two reasons may potentially account for the unfavorable performance when applying GRM to the Chinese NSTEMI patients. First, GRM was developed in patients mainly from America, Europe and Australia twenty years ago [11]. There are significant differences in characteristics and management between these and contemporary Chinese patients [3, 11]. Second, the GRM was developed to assess the risk of patients with acute coronary syndrome, including STEMI, NSTEMI, and unstable angina pectoris [11].

To address the limitations mentioned above, the CRM was developed to predict in-hospital mortality, particularly for Chinese NSTEMI patients, and its diagnostic performance was superior to that of the GRM in two internal validation studies using data from CAMI registry [8, 9]. Our effort further extended this conclusion in an external validation cohort by indicating a larger AUC, positive NRI, and less disagreement between the observed and predicted mortality for the CRM. Moreover, the GRM classified 15% (19/126) in-hospital dead into the low-risk category, for which invasive strategy was not recommend according to current guidelines and 22% (28/126) into the medium-risk category, for which immediate (<2 h) and early (<24 h) invasive strategy were not recommend [1]. Using the CRM instead of the GRM would imply a significant change for downstream therapeutic strategy in the in-hospital dead: 87% (36/47) of these patients would be reclassified into the higher risk category, for which more aggressive strategies were recommended. Thus, application of the CRM instead of the GRM may have the potential to optimize the referral of aggressive intervention and lead to an evident diminution of the treatment-risk paradox in NSTEMI patients.

This study was subjected to the limitations of its single-center, retrospective, and observational design. The indication of clinical management and downstream therapeutic strategy were based on the individual physician decision. Thus, the strategic discordance of downstream management for NSTEMI patients should not be ignored, and the actual impact of applying the CRM was complicated. To further investigate the generalizability and reliability of CRM-guided therapeutic strategy, more pragmatic and cost-effective randomized control trials are needed, such as the UK GRACE Risk Score Intervention Study (UKGRIS) [21].

5. Conclusions

In Chinese NSTEMI patients, the CRM provided a more effective estimation for in-hospital mortality due to the improvement in discrimination, classification, and calibration compared to the GRM. The application of the CRM instead of the GRM could change the diagnostic strategy and the potential to optimize the referral of a more aggressive intervention.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.


This study was funded by the Key Project of Scientific and Technological Support Plan of Tianjin in 2018 (No.16ZXMJSY00140).


  1. M. Roffi, C. Patrono, J.-P. Collet et al., “2015 ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation,” European Heart Journal, vol. 37, no. 3, pp. 267–315, 2016. View at: Publisher Site | Google Scholar
  2. E. A. Amsterdam, N. K. Wenger, R. G. Brindis et al., “2014 AHA/ACC guideline for the management of patients with non-ST-elevation acute coronary syndromes: a report of the American college of cardiology/American heart association task force on practice guidelines,” Journal of the American College of Cardiology, vol. 64, no. 24, pp. e139–e228, 2014. View at: Publisher Site | Google Scholar
  3. D. Zhao, J. Liu, M. Wang, X. Zhang, and M. Zhou, “Epidemiology of cardiovascular disease in China: current features and implications,” Nature Reviews Cardiology, vol. 16, no. 4, pp. 203–212, 2019. View at: Publisher Site | Google Scholar
  4. Q. Zhang, D. Zhao, W. Xie et al., “Recent trends in hospitalization for acute myocardial infarction in beijing: increasing overall burden and a transition from ST-segment elevation to non-ST-segment elevation myocardial infarction in a population-based study,” Medicine (Baltimore), vol. 95, no. 5, Article ID e2677, 2016. View at: Publisher Site | Google Scholar
  5. M. Hall, O. J. Bebb, T. B. Dondo et al., “Guideline-indicated treatments and diagnostics, GRACE risk score, and survival for non-ST elevation myocardial infarction,” European Heart Journal, vol. 39, no. 42, pp. 3798–3806, 2018. View at: Publisher Site | Google Scholar
  6. Q. Yang, Y. Wang, J. Liu et al., “Invasive management strategies and antithrombotic treatments in patients with non-ST-segment-elevation acute coronary syndrome in China: findings from the improving CCC Project (care for cardiovascular disease in China),” Circulation Cardiovascular Interventions, vol. 10, no. 6, 2017. View at: Publisher Site | Google Scholar
  7. A. Saar, T. Marandi, T. Ainla, K. Fischer, M. Blondal, and J. Eha, “The risk-treatment paradox in non-ST-elevation myocardial infarction patients according to their estimated GRACE risk,” International Journal of Cardiology, vol. 272, pp. 26–32, 2018. View at: Publisher Site | Google Scholar
  8. R. Fu, C. Song, J. Yang et al., “CAMI-NSTEMI score-China acute myocardial infarction registry-derived novel tool to predict in-hospital death in non-ST segment elevation myocardial infarction patients,” Circulation Journal: Official Journal of the Japanese Circulation Society, vol. 82, no. 7, pp. 1884–1891, 2018. View at: Publisher Site | Google Scholar
  9. C. Song, R. Fu, S. Li et al., “Simple risk score based on the China acute myocardial infarction registry for predicting in-hospital mortality among patients with non-ST-segment elevation myocardial infarction: results of a prospective observational cohort study,” BMJ Open, vol. 9, no. 9, Article ID e030772, 2019. View at: Publisher Site | Google Scholar
  10. H. Xu, W. Li, J. Yang et al., “The China Acute Myocardial Infarction (CAMI) Registry: a national long-term registry-research-education integrated platform for exploring acute myocardial infarction in China,” American Heart Journal, vol. 175, pp. 193–201.e3, 2016. View at: Publisher Site | Google Scholar
  11. C. B. Granger, R. J. Goldberg, O. Dabbous et al., “Predictors of hospital mortality in the global registry of acute coronary events,” Archives of Internal Medicine, vol. 163, no. 19, pp. 2345–2353, 2003. View at: Publisher Site | Google Scholar
  12. G. S. Collins, J. B. Reitsma, D. G. Altman, and K. G. M. Moons, “Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD group,” Circulation, vol. 131, no. 2, pp. 211–219, 2015. View at: Publisher Site | Google Scholar
  13. A. C. Alba, T. Agoritsas, M. Walsh et al., “Discrimination and calibration of clinical prediction models: users’ guides to the medical literature,” JAMA, vol. 318, no. 14, pp. 1377–1384, 2017. View at: Publisher Site | Google Scholar
  14. E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson, “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach,” Biometrics, vol. 44, no. 3, pp. 837–845, 1988. View at: Publisher Site | Google Scholar
  15. A. A. Kramer and J. E. Zimmerman, “Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited,” Critical Care Medicine, vol. 35, no. 9, pp. 2052–2056, 2007. View at: Publisher Site | Google Scholar
  16. M. J. Pencina, R. B. D’Agostino Sr., R. B. D’Agostino Jr., and R. S. Vasan, “Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond,” Statistics in Medicine, vol. 27, no. 2, pp. 157–172, 2008. View at: Publisher Site | Google Scholar
  17. A. W. Schoenenberger, D. Radovanovic, S. Windecker et al., “Temporal trends in the treatment and outcomes of elderly patients with acute coronary syndrome,” European Heart Journal, vol. 37, no. 16, pp. 1304–1311, 2016. View at: Publisher Site | Google Scholar
  18. D. P. Chew, G. Junbo, W. Parsonage et al., “Perceived risk of ischemic and bleeding events in acute coronary syndromes,” Circulation Cardiovascular Quality and Outcomes, vol. 6, no. 3, pp. 299–308, 2013. View at: Publisher Site | Google Scholar
  19. I. Ahrens, O. Averkov, E. C. Zuniga et al., “Invasive and antiplatelet treatment of patients with non-ST-segment elevation myocardial infarction: understanding and addressing the global risk-treatment paradox,” Clinical Cardiology, vol. 42, no. 10, pp. 1028–1040, 2019. View at: Publisher Site | Google Scholar
  20. W. Leng, J. Yang, X. Fan et al., “Contemporary invasive management and in-hospital outcomes of patients with non-ST-segment elevation myocardial infarction in China: findings from China Acute Myocardial Infarction (CAMI) Registry,” American Heart Journal, vol. 215, pp. 1–11, 2019. View at: Publisher Site | Google Scholar
  21. C. C. Everett, K. A. Fox, C. Reynolds et al., “Evaluation of the impact of the GRACE risk score on the management and outcome of patients hospitalised with non-ST elevation acute coronary syndrome in the UK: protocol of the UKGRIS cluster-randomised registry-based trial,” BMJ Open, vol. 9, no. 9, Article ID e032165, 2019. View at: Publisher Site | Google Scholar

Copyright © 2020 Peng Wang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Related articles

No related content is available yet for this article.
 PDF Download Citation Citation
 Download other formatsMore
 Order printed copiesOrder

Related articles

No related content is available yet for this article.

Article of the Year Award: Outstanding research contributions of 2021, as selected by our Chief Editors. Read the winning articles.