Abstract
Objective. To determine the accuracy of Point-Of-Care testing (PoCT) creatinine values when compared to standard central laboratory testing (IDMS) and to demonstrate if and how a discrepancy could lead to improper risk stratification for contrast induced nephropathy (CIN). Methods. We conducted a descriptive retrospective chart review of patients seen in the Emergency Department of a single suburban, community, and academic medical center. We included patients who presented to the department between March 2013 and September 2014 who had blood samples analyzed by both PoCT and IDMS. Results. Mean IDMS creatinine values were 0.23 mg/dL higher when compared with i-Stat values. 95% of the time, the IDMS creatinine value was variable and ranged from −0.45 mg/dL to +0.91 mg/dL when compared to the i-Stat creatinine. When using i-Stat creatinine values to calculate GFR, 47 out of 156 patients had risk category variations compared to using the IDMS value. This affected 30.1% of the total eligible sample population (22.9% to 37.3% with 95% CI). Conclusion. We found a significant discrepancy between PoCT and IDMS creatinine values and found that this discrepancy could lead to improper risk stratification for CIN.
1. Introduction/Background
Contrast-induced nephropathy (CIN) is a known complication of procedures in which intravascular iodinated contrast media are administrated [1, 2]. Patients with preexisting renal impairment have an increased risk of developing CIN. Previous studies have demonstrated that a glomerular filtration rate (GFR) <60 is a risk factor for CIN [3]. Creatinine value is an essential component when determining GFR. It is important to rapidly identify patients who have elevated creatinine to risk-stratify patients before administrating iodinated contrast media.
Currently, the gold standard to accurately measure creatinine in the emergency department (ED) is by isotope dilution mass spectrometry (IDMS) in a central laboratory. Ideally, this test can take at least thirty minutes for the laboratory to analyze. However, in reality, it can take even longer to obtain the value in a busy ED. During critical situations, obtaining the creatinine rapidly would allow the physician to determine the best diagnostic testing or therapy for a patient to prevent CIN. As an example, we feel that if we could use point-of-care testing (PoCT) rather than sending the samples to the central laboratory, this could expedite decision making in traumas and critical patients. PoCT with a blood analyzer such as the i-Stat system is being used to provide real-time results in the ED within minutes [2, 4].
If PoCT is as accurate as the IDMS from the central laboratory, using PoCT in these situations could allow the ED physician to quickly send a patient to CT scan for the appropriate study without delay.
During the past 18 months, our department obtained over 724 PoCT chemistry studies. Subjectively, many providers recall instances where the resulting i-Stat creatinine values were significantly different compared to the values from the central laboratory testing.
It is important to determine the accuracy of PoCT testing in real time since inaccurate PoCT testing will impact patient care. According to data published on their website, the PoCT system, the Chem8+, designed by Abbott, Point of Care, Inc., was verified independently and approved for use by the FDA in 2006. They verified their machine’s accuracy using linear regression analysis which demonstrated correlations all greater than 0.985. They also verified the precision of their system by testing multiple cartridges for 39 patient samples and showed the average creatinine difference across all reference ranges to be less than 0.04 mg/dL. According to their study, they noted specifically that acetaminophen, acetylcysteine, ascorbate, bromide, creatine, glycolic acid, and hydroxyurea could affect their Chem8+ values. This study was done in a controlled environment and we sought to determine if the results are still accurate in real-time use in an emergency department. If a physician uses an erroneous lab value for medical decision making, it could place a patient at higher risk for developing CIN or delay the physician from ordering an appropriate study.
2. Hypothesis
The creatinine measurement obtained via i-Stat PoCT does not equal the values obtained via IDMS standard laboratory testing.
3. Methods and Study Design
This study is a descriptive retrospective chart review of patients seen in the Emergency Department of a single suburban, community, academic medical center with an annual census of approximately 98,000 patients. The study includes patients who presented to the department from March 2013 (the implementation of our new EMR) to September 2014 who had blood samples analyzed by both i-Stat testing and the central laboratory IDMS. We investigated the measurement agreement between the PoCT i-Stat creatinine values and those from the central laboratory IDMS testing and how their discrepancy affected changes in the GFR risk categories.
In our department, respiratory therapists, who are certified by the central laboratory, perform PoCT. The PoCT used in this study is the Chem8+, which measures serum sodium, potassium, chloride, bicarbonate, anion gap, ionized calcium, glucose, urea nitrogen, hemoglobin, hematocrit, and creatinine. Since the creatinine level can change dynamically, we only included patients who had an i-Stat serum test performed within ten minutes of retrieving blood sent to the standard laboratory. Exclusion criteria included patients with missing or incomplete charts, patients not originally evaluated in the ED, and patients who had blood chemistries drawn outside of ten minutes of the i-Stat sample.
The data was abstracted from Abbott’s precision web point-of-care management system. All of the patients seen in the ED who had a PoCT i-Stat performed from March 2013 to September 2014 were enrolled. Their records were manually reviewed through our electronic medical record system (Epic System Hyperspace 2014 IU 1). The Epic and Abbott Precision Web system made it possible to include all eligible patients and analyze all data points including age, sex, weight, collection times, and sample type (arterial or venous). Once the patient population was obtained, the creatinine results via PoCT were compared to the creatinine results via standard laboratory testing. In addition, for data analysis purposes, we calculated the GFR for patients using the creatinine obtained from both samples using the Cockcroft-Gault formula [5], which is the formula our institution currently uses to accept patients for CT studies with IV contrast. Patients with a GFR <60 are not recommended to receive IV contrast studies. Private health information (PHI) was initially obtained but not stored in the study. The charts were only searched for collection times, sample type (arterial or venous), demographics, and creatinine value.
All statistical analysis was performed by use of a commercially available statistical program (Excel Microsoft 2008 and Excel Analyze-it 2007 Method Validation Edition).
4. Results
A total of 724 i-Stat creatinine tests were performed between March 2013 and September 2014 in the ED. The demographics are listed in Table 1. We excluded 568 samples from the study because of exclusion criteria. Of the 724 samples obtained over our study time frame, 156 patients were eligible for the study as shown in Figure 1.
All samples were obtained within 10 minutes of each other (median = 5 minutes; IQR = 2 to 8 minutes). Calculations for all of our samples concluded that, on average, IDMS samples had a creatinine value 0.23 mg/dL higher than PoCT samples. Only eight eligible patients had the same creatinine value for both samples. In 114 cases, the IDMS measurement was higher, and in 34 cases it was lower than the i-Stat PoCT value (as shown in Table 2).
In this study, we are interested in the accuracy of PoCT creatinine testing compared to IDMS creatinine. We used a Bland-Altman plot [6, 7] to analyze the agreement between the two values. The Bland-Altman method plot demonstrates the agreement between two methods of measurement by plotting the hypothetical true value (average of the two methods) against the difference of the two measurements. Figure 2 demonstrates the agreement between PoCT i-Stat creatinine values and the IDMS values in our study.
The plot shows a trend of positive values for IDMS creatinine and i-Stat creatinine. This implies that IDMS creatinine tends to have higher values. Also we can see how most outliers stand on the extremes of the difference.
The central laboratory creatinine values had a measurement interval of 0.4 mg/dL to 15 mg/dL, while those of the i-Stat testing had a measurement interval of 0.1 mg/dL to 15.8 mg/dL. The laboratory creatinine average was 1.84 mg/dL while the i-Stat creatinine average was 1.61 mg/dL. The Bland-Altman test found that the mean value of the difference between the two measurements is 0.23 mg/dL and since it is different than a perfect agreement of 0, it is considered a fixed bias between two methods. The limits of agreement were calculated as the bias ±1.96 standard deviations of 0.35, which calculates the difference value between the creatinine obtained by both measurements to range from −0.45 mg/dL to 0.91 mg/dL. This range accounts for fixed bias as well as error measurement. Accuracy was defined as the 95% confidence interval of the limits of agreement, which corresponds with statistical normal distribution avoiding measurement error.
The patients’ GFRs were calculated according to the Cockcroft-Gault equation. These values were then categorized according to the levels of risk following the Guidelines for Contrast Administration from the American College of Radiology [8–13]. The four categories consist of high, intermediate, low, and negligible risk. The high risk category includes GFR values less than 30 mL/min/1.73 m2; the intermediate risk category includes GFR values between 30 and 44 mL/min/1.73 m2; the low risk category includes GFR values between 45 and 60 mL/min/1.73 m2; the negligible risk category includes GFR values higher than 60 mL/min/1.73 m2. We then calculated the number of patients whose level of risk was not accurately predetermined when their GFR was calculated using the PoCT creatinine value versus their IDMS value. Even though most of the patients’ creatinine changed, the level of risk remained in the same category for 109 patients, 9 patients’ risk category decreased, and 38 patients’ risk category increased when using their i-Stat creatinine value versus their IDMS creatinine value. Thirty-eight out of those 47 patients (80.9%, 99% CI 66.1 to 95.6%) were actually at a higher risk category for CIN. This data is displayed in Tables 3(a) and 3(b).
5. Discussion
Physicians in our ED have the ability to obtain PoCT creatinine values. These results are often available much sooner than standard IDMS testing, which must be sent to the hospital’s central laboratory. Creatinine is a well-established marker for renal function and it is widely used to estimate the GFR [14]. GFR is used to risk-stratify patients for their ability to tolerate IV contrast for CT scans. Often, IV contrast CT scans are the ideal study to rule out significant pathology in critically ill patients. Therefore, obtaining an accurate value of a patient’s creatinine is imperative to the physician’s diagnosis. There were many anecdotal reports from the ED physicians in our ED noting discrepancies between creatinine values obtained via PoCT testing and IDMS testing. Our study analyzed if in fact there are significant differences in values and if the values altered the patients’ risk stratification.
Our data analysis revealed the following.
(1) Based upon our results, 95% of the time, the IDMS creatinine value was variable and could be either 0.45 mg/dL below or 0.91 mg/dL above the i-Stat creatinine. In our emergency department, on average, IDMS creatinine is 0.23 mg/dL higher than the one from the i-Stat testing.
Calculations reveal an average of 22% difference between IDMS and the i-Stat creatinine values, which is significantly higher than the desirable analytical goal of 8.87% total error for creatinine serum as indicated by Westgard [14]. This variability, fixed bias, and percentage of difference limit the ability of the physicians in our ED to make a reliable, well-informed decision on whether or not to order a CT scan with contrast.
(2) When using i-Stat creatinine values to calculate GFR, 47 out of 156 patients had risk category variations. Those are equal to 30.1% of total eligible sample population (22.9% to 37.3% with 95% CI). 38 out of those 47 patients (80.9%, 99% CI 66.1 to 95.6%) were actually at a higher risk category for CIN.
For the patients eligible for inclusion in our study, almost one-fourth of them would have increased risk for IV contrast-induced nephropathy if their GFRs were calculated using creatinine values from IDMS testing versus those from PoCT testing. This is a significant cause for concern and places a significant percentage of patients at risk.
In addition, if GFRs were calculated using PoCT creatinine values, rather than IDMS creatinine values, 9 out of 156 patients (6%) may have been eligible for IV contrast based on GFR risk stratification and not received these studies due to underestimation of their GFR. This could lead to a potential delay in obtaining the ideal radiologic study and potentially lead to repeat imaging with IV contrast, which exposes the patient to additional radiation. Therefore, basing GFR calculations on i-Stat creatinine in our ED could mean wrongly risk-stratifying the patients.
We know that neither of the previously mentioned scenarios is acceptable in our ED. Qualitatively our study demonstrates that our ED physicians should not use only PoCT creatinine values to risk-stratify patients for CT imaging studies if creatinine values are within the variation range.
Every six months, i-Stat receives validations by two regulatory agencies, JACHO and NYS. The FDA has reviewed i-Stat Chem8+ and CLEA has waived it [15]. However, the tests were only run on potassium. In addition, Abbott participates in an external proficiency testing three times a year, even though they only need an 80% match to pass. The i-Stat manual describes a thorough process to generate creatinine values [16]. We know that i-Stat PoCT is a highly convenient device and we know that there is a high correlation of 0.98 between the two measurements. However, in our ED, the variation is higher than desirable and therefore caution should be used for decision making when relying on the PoCT.
Overall, this study is concerning for the accuracy of PoCT creatinine testing. Our results seem to demonstrate a difference from the published data for accuracy from Abbott. It is unclear what is causing the difference in creatinine values. Potential causes include therapist technique, type of syringes used, venous versus arterial samples, inaccurate calibration by the hospital laboratory, and test inaccuracy. Future follow-up studies are needed to investigate the source of these inaccuracies and offer possible suggestions on improving such a convenient and needed device as the i-Stat in EDs. Our study also did not look at the accuracy of other values obtained on the i-Stat such as hemoglobin, hematocrit, and potassium or the repeatability of the measurements for the same patients. We hope that a future study will look into the accuracy of those values as well.
6. Summary
In conclusion, we analyzed the accuracy of PoCT creatinine testing compared to IDMS creatinine. We found significant discrepancy between the two methods of measurement in our controlled sample population. The clinical significance of these discrepancies should be taken into consideration when point-of-care testing is performed. There may be a potential to misclassify patient risk when utilizing PoCT.
Limitations
This study has multiple limitations. It is limited by being based at a single center, which is a suburban, community based, and academic emergency department. The results from this study are not necessarily representative of other centers. The study was limited to a single PoCT device used in our ED. The results are also limited by the techniques of the respiratory therapists who performed the studies. We also only had 156 of 724 patients who met all inclusion criteria. There are limitations in calculating the GFR because of patient weight. We assume all weights to be accurate. However, many weights in our department are estimations. It is unclear if this would have any significant difference on the true change in GFR. Our study is also limited by the fact that most patients in our department, who have i-Stats obtained, are critically ill patients and the results may not be representative of the overall demographics of our department. This study was also limited by the fact that we did not look at outcomes as correlated with the GFR and the study was underpowered to look at correlations of discrepancies with patient age. Another assumption made in this study is that we hold the laboratory as the standard measure and consider it to reveal the true value of creatinine. However, it was possible that the laboratory’s calibration methods lacked the required degree of accuracy. This study did not attempt to quantify the actual risk to patients as there is no universal quantifiable risk identified. The study was not designed to find trends on admissions based on GFR.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.