Evaluation of the Canadian Clinical Practice Guidelines Risk Prediction Tool for Acute Aortic Syndrome: The RIPP Score

Ohle, Robert; McIsaac, Sarah; Van Drusen, Madison; Regis, Aaron; Montpellier, Owen; Ludgate, Mackenzie; Bodunde, Oluwadamilola; Savage, David W.; Yadav, Krishan

doi:https://doi.org/10.1155/2023/6636800

Emergency Medicine International

On this page

Abstract Introduction Methods Results Discussion Conclusion Data Availability Conflicts of Interest Authors’ Contributions Supplementary Materials References Copyright Related Articles

Research Article | Open Access

Volume 2023 | Article ID 6636800 | https://doi.org/10.1155/2023/6636800

Evaluation of the Canadian Clinical Practice Guidelines Risk Prediction Tool for Acute Aortic Syndrome: The RIPP Score

Robert Ohle,¹Sarah McIsaac,²Madison Van Drusen,³Aaron Regis,⁴Owen Montpellier,⁴Mackenzie Ludgate,⁴Oluwadamilola Bodunde,⁴David W. Savage,⁵and Krishan Yadav⁶

Academic Editor: Guangtao Xu

Received07 Dec 2022

Revised24 Mar 2023

Accepted04 May 2023

Published25 May 2023

Abstract

Introduction. Acute aortic syndrome (AAS) is a rare clinical syndrome with a high mortality rate. The Canadian clinical practice guideline for the diagnosis of AAS was developed in order to reduce the frequency of misdiagnoses. As part of the guideline, a clinical decision aid was developed to facilitate clinician decision-making (RIPP score). The aim of this study is to validate the diagnostic accuracy of this tool and assess its performance in comparison to other risk prediction tools that have been developed. Methods. This was a historical case-control study. Consecutive cases and controls were recruited from three academic emergency departments from 2002–2020. Cases were identified through an admission, discharge, or death certificated diagnosis of acute aortic syndrome. Controls were identified through presenting complaint of chest, abdominal, flank, back pain, and/or perfusion deficit. We compared the clinical decision tools’ C statistic and used the DeLong method to test for the significance of these differences and report sensitivity and specificity with 95% confidence intervals. Results. We collected data on 379 cases of acute aortic syndrome and 1340 potential eligible controls; 379 patients were randomly selected from the final population. The RIPP score had a sensitivity of 99.7% (98.54–99.99). This higher sensitivity resulted in a lower specificity (53%) compared to the other clinical decision aids (63–86%). The DeLong comparison of the C statistics found that the RIPP score had a higher C statistic than the ADDRS (−0.0423 (95% confidence interval −0.07–0.02); ) and the AORTAs score (−0.05 (−0.07 to −0.02); P = 0.0002), no difference compared to the Lovy decision tool (0.02 (95% CI −0.01–0.05 )) and decreased compared to the Von Kodolitsch decision tool (0.04 (95% CI 0.01–0.07 )). Conclusion. The Canadian clinical practice guideline’s AAS clinical decision aid is a highly sensitive tool that uses readily available clinical information. It has the potential to improve diagnosis of AAS in the emergency department.

1. Introduction

Acute aortic syndrome is a time-sensitive emergency defined by three distinct diagnoses: aortic dissection, intramural hematoma, and penetrating atherosclerotic ulcer. Acute aortic syndrome and each of its component diagnoses involve blood leaking into the wall of the aorta, which is the major artery that supplies blood to the entire body [1]. This blood separates the layers of the wall of the aorta and can block blood flow to vital organs (heart, muscle, brain, or limbs) or lead to a rupture of the aorta and catastrophic blood loss [2–5]. The risk of death from acute aortic syndrome increases by 2% per hour, reaching 90% if undiagnosed [6]. Currently, there is significant variation in how physicians investigate acute aortic syndrome. This variation has led to inefficient use of computed tomographic imaging and a high miss rate. Health care providers miss 1 in 4 cases on the first presentation [2, 7–9]. Computed tomography is the investigation of choice to diagnose acute aortic syndrome. Its use in the emergency department has increased exponentially over the past number of years without any impact on the number of missed cases [10].

We have developed the Canadian clinical practice guideline for the diagnosis of acute aortic syndrome in the emergency department [11]. The goal of this guideline is to offer practical recommendations to guide the investigation of acute aortic syndrome, reducing misdiagnosis and improving efficiency of resource utilization. The guideline committee proposed a consensus-based risk stratification tool to aid in pretest probability assessment for those at risk of acute aortic syndrome (RIPP score shown in Table 1).

The aim of this study is to validate the diagnostic accuracy of this tool and assess its performance in comparison to other risk prediction tools that have been developed.

2. Methods

This was a historical case-control study. Consecutive cases were recruited from three academic emergency departments from 2002–2020, with an annual census ranging from 75,000–120,000. Consecutive controls were recruited from a single academic emergency department. These departments were geographically distributed across Ontario, with one northern and two southern sites. This study follows the methodological and reporting recommendations outlined in the Standards for eporting of Diagnostic Accuracy Studies (STARD) criteria. [12] The study was approved by the research ethics boards of all participating institutions.

2.1. Data Extraction

The data were extracted and verified from multiple sources, including emergency department records and progress notes. Trained reviewers used electronic data forms and underwent training [13]. The data were analysed for agreement using the Kappa statistic with oversight. Interobserver agreement was calculated for at least 20% of the total charts. Reviewers were not informed of the study objective but were aware of the patient’s case or control status.

2.2. Variables

The full data dictionary is included in the appendix. Abrupt onset pain was defined as pain described as sudden, unexpected, or onset at a specific time point. Severe was a pain score greater than 6 or required opioids for pain relief [14, 15]. Migrating/radiating pain was defined as pain that moved from one location to another or pain reported in two distinct anatomical locations, i.e., chest and back. Hypotension was defined by a systolic blood pressure <90 mmHg [15]. The clinical suspicion variable had three levels: AAS as most likely diagnosis, alternative diagnosis as more likely, and unsure. AAS was considered the most likely diagnosis if, after the initial assessment, a CT aorta was ordered to rule out AAS. An alternative diagnosis was considered most likely if documentation of a specific discharge or admission diagnosis was present. The clinical impression was deemed unsure if the diagnosis was unspecified or not yet diagnosed (i.e., chest pain not yet diagnosed).Where a score offered three risk levels, we dichotomized them into high risk and low risk.

2.3. Participants

2.3.1. Case

We included consecutive eligible cases of acute aortic syndrome from 2002 to 2020. Cases were identified through an admission, discharge, or death certificated diagnosis of acute aortic syndrome.

2.3.2. Control

Controls were identified through the presenting complaint of chest, abdominal, flank, back pain, and/or perfusion deficit (limb ischemia, cerebrovascular accident possible, neurological deficit, syncope, or altered level of consciousness). These were based on the Canadian Emergency Department Information System (CEDIS), which presents complaints that are indexed in our electronic health records. We randomly selected an equal number of unmatched controls (1 : 1) to be included in the final population.

2.3.3. Exclusion

We excluded patients <18 years old, with a pain duration of >14 days, or with trauma within 24 hours of the onset of pain. No documented chest, abdomen, back pain, or perfusion deficit. Left without being seen or with no documentation available for the patient. A clear alternative diagnosis after initial clinical assessment, i.e., subcutaneous abscess, urinary tract infection, trapezius muscle strain, panic attack, cannabis hyperemesis syndrome, gastroenteritis, upper respiratory tract infection, uterine prolapse, upper gastrointestinal bleed, etc.

2.3.4. Outcome Measures

Radiological evidence of aortic dissection, intramural hematoma, or penetrating atherosclerotic ulcer was used to identify acute aortic syndrome. The absence of acute aortic syndrome was verified through imaging. For patients who were not imaged, their medical records were reviewed for up to six months postencounter to ensure that no new diagnosis of acute aortic syndrome was made. Follow-up visits to hospitals without a diagnosis of acute aortic syndrome or imaging that did not reveal acute aortic syndrome were used to confirm the absence of the condition. In cases where patients did not return to study hospitals or undergo further imaging, publicly available sources such as obituaries were searched to determine if they had passed away.

2.3.5. Data Analyses

We calculated the classification performance of the RIPP score using sensitivity and specificity together with 95% confidence intervals. We also assessed the classification of the acute aortic dissection detection risk score (ADDRS), the Lovy clinical decision tool, the Von Kodolitsch decision tool, and the AORTAs score according to their ability to classify patients as either low or high risk [16–19]. We compared the clinical decision tools’ C statistic (area under the curve) and used the DeLong method to test for the significance of these differences. We calculated the absolute net reclassification indices by comparing the RIPP score with the other clinical decision tools [20]. The net reclassification index quantifies the improvement in prediction when comparing prediction tools. The sample size was based on an estimation of the precision of the classification performance of the risk scale. Our goal was to ascertain sufficient cases to evaluate the sensitivity with 95% confidence bands plus/minus 5%, corresponding to 200 cases of acute aortic syndrome.

2.3.6. Patient and Public Involvement

Neither patients nor the public were formally involved in the planning of the study. We plan to involve patients before assessing the effects of implementing this rule in clinical practice.

3. Results

Data were collected from 2002 to 2021, yielding 379 cases of acute aortic syndrome. We found 1340 potential eligible controls over a 1-month period; 580 were excluded. Of the remaining controls, 379 patients were randomly selected for the final population (Figure 1).

The Kappa between data extractors after chart training was 0.87. Chest pain unspecified (20.7%), abdominal pain unspecified (9.9%), and acute coronary syndrome (8.7%) were the top three diagnoses in the control population. Table 2 shows the clinical features of our cohort. The patients had a mean age of 68.5 years, and 52.3% were female.

The RIPP score had a sensitivity of 99.7%. This was higher than all the other scores. This higher sensitivity resulted in a lower specificity (53%) compared to the other clinical decision aids (63–86%) (Table 3). The DeLong comparison of the C statistics found the RIPP score had a higher C statistic compared to the ADDRS (−0.0423 (95% confidence interval −0.07–0.02); ) and the AORTAs score (−0.05 (−0.07 to −0.02); P = 0.0002), no difference compared to the Lovy decision tool (0.02 (95% CI −0.01–0.05 )), and decreased compared to the Von Kodolitsch decision tool (0.04 (95% CI 0.01–0.07 )) (Figure 2. Table 4).

Table 5 shows the absolute net reclassification index between the RIPP score and the other clinical decision tools. This ranged from 0.03 compared to the AORTAs to 0.20 for the Von Kodolitsch score. The RIPP score had the highest number of cases of acute aortic syndrome that were correctly classified as high-risk.

4. Discussion

Our study provisionally validates the predictive performance of the RIPP score in a broad sample of patients identified in the emergency department with a diagnosis of acute aortic syndrome. To improve the generalisability of the score, we included both large and smaller volume centres, including sites that were not involved in the derivation of any of the assessed scores. The score was able to correctly stratify many more patients with acute aortic syndrome into an appropriate high-risk category. This rule meets the prespecified sensitivity identified by emergency medicine clinicians to successfully be used in guiding investigations for patients with acute aortic syndrome [21].

It should be noted that this is a retrospective case-control study and suffers from significant bias in the assessment of diagnostic accuracy estimates. This bias should affect all scores equally and therefore be nondifferential. However, this study only provides provisional evidence of diagnostic accuracy; it is not sufficient in isolation to change practice.

4.1. Comparison with Other Studies

We had previously surveyed emergency physicians to identify thresholds of acute aortic syndrome risk that would alter clinical decisions. In this study, respondents indicated that patients with a subsequent risk of acute aortic syndrome below 1% were most appropriate for no further investigation [21].

In our study, the RIPP score was the only decision tool that had sufficiently high sensitivity to be able to define a no testing group. The ADDRS and the AORTA scores are designed to be used with D-dimer and allow the user to differentiate between D-dimer versus computed tomography as the first test in an investigation [14, 19]. Our study confirms that they are unable to identify a low-risk group that requires no further testing, and their use should be confined to assessing D-dimer versus computed tomography as the initial investigation. It was expected that there would be a high level of missing data for D-dimer within our cohort; therefore, a priori, we decided to assess each decision aids ability to define a no testing cohort.

The Lovy and Von Kodolitsch decision aids both use clinical variables to define a low- or a high-risk group of patients. To our knowledge, this is the first evaluation of these decision aids. We found that neither had sufficient sensitivity to define a no testing group.

4.2. Limitations

The case-control design of this study does not allow us to define the number of selected cases and control participants relative to the source population. Therefore, baseline risks, hazards, and absolute probabilities cannot be correctly adjusted. However, the risk of bias is applicable to all decision tools tested, and the comparative differences are not affected.

One variable of the score was clinical impression. This was extracted from the patient chart; therefore, a physician may have had a clinical suspicion for AAS but not documented the diagnosis. This may artificially inflate the specificity of the clinical risk score. We were conservative in our estimate of the presence of a clinical impression for AAS, only coding as yes if a CT aorta was ordered to rule out AAS; therefore, the inherent fragility of this variable should not affect the reported sensitivity.

Our aim was to select a control population that reflected those in whom there would be a reasonable consideration of acute aortic syndrome. We excluded those who had an image-proven alternative diagnosis or those who underwent computed tomography with intravenous contrast for another reason and would have been diagnosed with acute aortic syndrome. Because of this, we were left with a population that was less severe in presentation than those who underwent imaging for other reasons. This likely artificially inflates our specificity; however, the comparative differences in specificity between tools should be independent of this bias.

This was a retrospective chart review where missing values were defaulted to negative. This was done to bias our results towards the lowest possible sensitivity. In addition, no assessment of the interrater reliability of the clinical variables could be performed.

Our cases are patients in whom acute aortic syndrome was identified; patients who were never identified, discharged, or died will not be identified. These patients are likely to have a different presentation, and thus the RIPP score may not perform as well in this patient population.

Not all patients underwent the reference standard of computed tomography, MRI, or transesophageal echocardiography. Therefore, there is a possibility of misclassifying cases as controls. This could artificially increase our sensitivity and specificity. We could potentially have chosen a control population from patients who underwent imaging to rule out acute aortic syndrome; however, that would have resulted in partial verification bias and artificially lowered the specificity.

4.3. Research and Clinical Implications

A prospective multicentre implementation study following established implementation guidelines is now needed to assess the impact of the RIPP score when applied in clinical practice. Although a tool may have sufficient accuracy to classify cases and controls, it still may not lead to the desired decrease in missed cases of AAS or the efficient use of advanced imaging.

The major concern with all of the clinical risk scores is the likely low specificity in clinical practice [22]. We found that the RIPP score had the lowest specificity. In order for the RIPP score to achieve sufficient sensitivity to meet prespecified criteria, there is a corresponding increase in false positives. The question remains: is the potential decrease in the missed rate of this rare diagnosis adequately balanced with the likely increase in CT usage? Although clinicians may use the score to help guide them in the collection of important clinical variables to help define risk for AAS, caution should be used until an implementation study is completed in order to confirm its use leads to better patient outcomes. Clinical decision rules and risk scores, when implemented, may not lead to a clinically important change in patient-oriented outcomes [22].

5. Conclusion

The RIPP score identifies those who require further investigation for acute aortic syndrome among those presenting to the emergency department with undifferentiated symptoms. Incorporating this validated clinical decision tool into clinical practice has the potential to improve decisions regarding the investigation of patients for acute aortic syndrome in the emergency department.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon reasonable request.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Authors’ Contributions

Dr. Ohle was responsible for study design, analysis, and manuscript preparation. Ms. Van Drusen, Mr. Regis BSc⁴, Mr. Montpellier BSc ⁴, Mr. Ludgate ⁴, and Mr. Bodunde were responsible for data extraction. Dr. Yadav and Dr. McIsaac were responsible for manuscript editing, formatting, cowriting, and methodological support. Dr. Ohle acts as guarantor for accuracy and integrity of the manuscript.

Supplementary Materials

Appendix-Data dictionary with concise definitions used for data extraction. (Supplementary Materials)

References

L. A. Pape, M. Awais, E. M. Woznicki et al., “Presentation, diagnosis, and outcomes of acute aortic dissection: 17-year trends from the International Registry of Acute Aortic Dissection,” Journal of the American College of Cardiology, vol. 66, no. 4, pp. 350–358, 2015.
View at: Publisher Site | Google Scholar
K. M. Harris, C. E. Strauss, K. A. Eagle et al., “Correlates of delayed recognition and treatment of acute type A aortic dissection the international registry of acute aortic dissection (IRAD),” Circulation, vol. 124, no. 18, pp. 1911–1918, 2011.
View at: Publisher Site | Google Scholar
M. Chua, I. Ibrahim, X. Neo, V. Sorokin, L. Shen, and S. B. Ooi, “Acute aortic dissection in the ED: risk factors and predictors for missed diagnosis,” The American Journal of Emergency Medicine, vol. 30, no. 8, pp. 1622–1626, 2012.
View at: Publisher Site | Google Scholar
S. Zhan, S. Hong, L. Shan-shan et al., “Misdiagnosis of aortic dissection: experience of 361 patients,” Journal of Clinical Hypertension, vol. 14, no. 4, pp. 256–260, 2012.
View at: Publisher Site | Google Scholar
K. Hirata, M. Wake, T. Takahashi et al., “Clinical predictors for delayed or inappropriate initial diagnosis of type A acute aortic dissection in the emergency room,” PLoS One, vol. 10, no. 11, Article ID e0141929, 2015.
View at: Publisher Site | Google Scholar
P. G. Hagan, C. A. Nienaber, E. M. Isselbacher et al., “The international registry of acute aortic dissection (IRAD): new insights into an old disease,” JAMA, vol. 283, no. 7, pp. 897–903, 2000.
View at: Publisher Site | Google Scholar
P. R. Sullivan, A. B. Wolfson, R. D. Leckey, and J. L. Burke, “Diagnosis of acute thoracic aortic dissection in the emergency department,” The American Journal of Emergency Medicine, vol. 18, no. 1, pp. 46–50, 2000.
View at: Publisher Site | Google Scholar
M. S. Hansen, G. J. Nogareda, and S. J. Hutchison, “Frequency of and inappropriate treatment of misdiagnosis of acute aortic dissection,” The American Journal of Cardiology, vol. 99, no. 6, pp. 852–856, 2007.
View at: Publisher Site | Google Scholar
I. Asouhidou and T. Asteri, “Acute aortic dissection: be aware of misdiagnosis,” BMC Research Notes, vol. 2, no. 1, p. 25, 2009.
View at: Publisher Site | Google Scholar
J. K. Takaki, I. Ford, and H.-C. Yoon, “Variation in CTA evaluation of ED patients suspected of aortic dissection,” Emergency Radiology, vol. 29, no. 4, pp. 709–713, 2022.
View at: Publisher Site | Google Scholar
R. Ohle, J. Yan, K. Yadav et al., “Canadian clinical practice guidelines for the diagnosis of acute aortic syndrome in the emergency department,” Canadian Medical Association Journal, vol. 192, 2020.
View at: Google Scholar
P. M. Bossuyt, J. B. Reitsma, D. E. Bruns et al., “Stard 2015: an updated list of essential items for reporting diagnostic accuracy studies,” Radiology, vol. 277, no. 3, pp. 826–832, 2015.
View at: Publisher Site | Google Scholar
A. C. Jansen, E. S. van Aalst-Cohen, B. A. Hutten, H. R. Büller, J. J. Kastelein, and M. H. Prins, “Guidelines were developed for data collection from medical records for use in retrospective analyses,” Journal of Clinical Epidemiology, vol. 58, no. 3, pp. 269–274, 2005.
View at: Publisher Site | Google Scholar
P. Nazerian, C. Mueller, A. de Matos Soeiro et al., “Diagnostic accuracy of the aortic dissection detection risk score plus D-dimer for acute aortic syndromes: the ADvISED Prospective Multicenter Study,” Circulation, vol. 117, Article ID 029457, 2017.
View at: Google Scholar
R. Ohle, J. W. Yan, K. Yadav et al., “Diagnosing acute aortic syndrome: a Canadian clinical practice guideline,” Canadian Medical Association Journal, vol. 192, no. 29, pp. E832–E843, 2020.
View at: Publisher Site | Google Scholar
L. F. Hiratzka, G. L. Bakris, J. A. Beckman et al., “2010 ACCF/AHA/AATS/ACR/ASA/SCA/SCAI/SIR/STS/SVM guidelines for the diagnosis and management of patients with thoracic aortic disease,” Journal of the American College of Cardiology, vol. 55, no. 14, pp. e27–e129, 2010.
View at: Publisher Site | Google Scholar
A. J. Lovy, E. Bellin, J. M. Levsky, D. Esses, and L. B. Haramati, “Preliminary development of a clinical decision rule for acute aortic syndromes,” The American Journal of Emergency Medicine, vol. 31, no. 11, pp. 1546–1550, 2013.
View at: Publisher Site | Google Scholar
Y. von Kodolitsch, A. G. Schwartz, and C. A. Nienaber, “Clinical prediction of acute aortic dissection,” Archives of Internal Medicine, vol. 160, no. 19, pp. 2977–2982, 2000.
View at: Publisher Site | Google Scholar
F. Morello, P. Bima, E. Pivetta et al., “Development and validation of a simplified probability assessment score integrated with age-adjusted d-dimer for diagnosis of acute aortic syndromes,” Journal of the American Heart Association, vol. 10, no. 3, Article ID e018425, 2021.
View at: Publisher Site | Google Scholar
M. J. Pencina, R. B. D’Agostino, R. B. D’Agostino, and R. S. Vasan, “Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond,” Statistics in Medicine, vol. 27, no. 2, pp. 157–172, 2008.
View at: Publisher Site | Google Scholar
R. Ohle, J. Yan, K. Yadav et al., “National survey of emergency physicians on the risk stratification and acceptable miss rate of acute aortic syndrome,” Canadian Journal of Emergency Medicine, vol. 22, 2020.
View at: Google Scholar
R. Ohle, O. Anjum, H. Bleeker, and S. McIsaac, “What is the specificity of the aortic dissection detection risk score in a low-prevalence population?” Academic Emergency Medicine: Official Journal of the Society for Academic Emergency Medicine, vol. 26, no. 6, pp. 632–638, 2019.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2023 Robert Ohle et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

467

Downloads

356

Citations