Estimation of Insulin Resistance in Mexican Adults by the [13C]Glucose Breath Test Corrected for Endogenous Total CO2 Production
Objective. To evaluate the efficacy of the [13C]glucose breath test for measuring insulin resistance in Mexican adults with different glycemic states. Research Design and Methods. Fifty-eight adults underwent a [13C]glucose breath test with simultaneous measurement of total CO2 production by indirect calorimetry, at baseline and 90 minutes after the ingestion of 15 g of dextrose and 25 mg of [13C]glucose. HOMA was used as a marker of insulin resistance. Results. We found an inverse correlation between HOMA and the breath test δ13CO2 (‰), (). After adjusting for total CO2 production, correlations between HOMA and fasting glucose were less strong but remained significant. An ROC curve was constructed using δ13CO2 (‰) and HOMA values; the cut-off point was 9.99‰ δ13CO2, corresponding to a sensitivity of 80.0 (95% CI: 51.9, 95.7) and a specificity of 67.4 (95% CI: 51.5, 80.9). Conclusions. The [13C]glucose breath test is a simple noninvasive procedure but was not sufficiently robust for an accurate diagnosis of insulin resistance. Our findings suggest that the test might be helpful in identifying individuals who are not IR, which in turn may contribute to improved diabetes prevention.
Obesity represents a major risk in the development of several chronic conditions, such as type 2 diabetes, hypertension, dyslipidemia, and atherosclerotic cardiovascular diseases. The precise molecular pathways underlying this close association are only partially understood [1, 2]. Type 2 diabetes is commonly preceded by a preclinical state known as pre-diabetes , which involves subclinical β-cell dysfunction known as insulin resistance (IR) . Insulin resistance has been recognized as the key factor in the development of type 2 diabetes [5–7]. Moreover, data from clinical trials have shown that IR can be reduced by weight loss and changes in lifestyle behaviors and that early interventions to reduce IR can delay or prevent the onset of type 2 diabetes [8–12]. Thus, there is a need to identify individuals with IR before they develop glucose intolerance and type 2 diabetes; various methods are available, but the practical diagnosis of IR remains a challenge.
The gold standard for IR measurement is the hyperinsulinemic-euglycemic clamp [13, 14]. This method is highly sensitive; however, it requires several blood samples is invasive, expensive, time consuming, and impractical for clinical use . Therefore, new simpler methods have been developed to identify stages of prediabetes, such as the homeostasis model assessment (HOMA)  and the quantitative insulin sensitivity check index (QUICKI) . More recently, glycated hemoglobin (HbA1c) has been proposed as a diagnostic test . The results of these methods have shown a strong correlation with IR as measured by the gold standard method and the oral glucose tolerance test (OGTT) [15, 17].
A novel noninvasive method has also been proposed, involving estimation of IR based on carbohydrate metabolism of 13C-labeled glucose . In healthy individuals, insulin promotes glucose uptake by cells to be metabolized. As a product of metabolism, CO2 is generated and subsequently removed by the lungs through breathing . Thus, when a dose of [13C]glucose is orally administered, the isotope tracer can be followed and quantified from a breath sample. Individuals with insulin resistance or type 2 diabetes show impaired glucose uptake and accordingly a lesser amount of expired 13CO2.
Several studies have demonstrated a good correlation between the results of the [13C]glucose breath test and the hyperinsulinemic-euglycemic clamp as well as with the surrogate techniques HOMA and QUICKI, in adults and children [18, 20, 21]. On the other hand, some authors have stated that a correct estimation of 13CO2 is only possible if total CO2 production is measured in parallel to the breath test , especially for IR estimation. In our review of the literature, we did not find publications where a [13C]glucose breath test included the measurement of total CO2 production.
We hypothesized that ingestion of [13C]glucose would produce an increase in expired 13CO2, which would provide a direct measure of insulin sensitivity, showing significant correlation with HOMA in Mexican individuals with different glycemic states. The aim of this study was to evaluate the efficacy of the [13C]glucose breath test to measure IR, considering total CO2 production in an adult Mexican population.
2. Research Design and Methods
This study was approved by the Ethics Committee of the Research Center for Food and Development (CIAD: Centro de Investigación en Alimentación y Desarrollo). All participants gave written informed consent before starting the protocol. Sample size was calculated based upon Pearson’s correlation coefficient for association of quantitative variables, given a value of and a power of 80% (1-β) . The sample number was 63, but only 58 remained after excluding 2 subjects for other medical problems and 3 dropouts. This reduced the power in the order of 15%; however, the association remained highly significant. This sample was selected to potentially represent different glycemic states; 58 adult women and men (>18 years old) were allocated into three groups: healthy non-obese individuals (BMI 18.5–24.9), overweight and obese individuals (), and newly diagnosed type 2 diabetes patients without medical treatment for diabetes [24, 25]. All subjects underwent a medical history review and physical examination by a physician to confirm eligibility. Criteria for exclusion included clinical evidence of other disorders linked to insulin resistance, such as thyroid dysfunction, acromegaly, pheochromocytoma, Cushing syndrome, and primary aldosteronism, pregnancy, and the use of drugs that could alter carbohydrate metabolism.
Subjects were asked not to perform exercise the day before the measurements and also to avoid the use of alcohol and caffeine. A venous blood sample was obtained from every subject after a 12-hour fast. Dietary intake was not determinate in participating subjects. Serum glucose and insulin were measured to calculate HOMA-IR according to Matthews et al. . Insulin was measured by immunoassay (ARCHITECT Insulin, Germany). Plasma glucose, total cholesterol (TC), triglycerides (TG), and HDL and LDL cholesterol were measured immediately after the blood sample was obtained by standard methods from Randox. Estimation of HbA1c was conducted by immunoaffinity (Randox). Blood pressure (BP) was measured using an aneroid sphygmomanometer (Welch Allyn Tycos, NY, USA) according to the guidelines given by the American Heart Association .
Body weight (BW) was measured with subjects dressed in light clothing using an OHAUS digital electronic scale (with a capacity of kg). Standing height (Ht) was measured to the nearest millimeter with a Holtain stadiometer (with a capacity of cm), Holtain Limited, Dyfed, UK. Body mass index (BMI, kg/m2) was calculated based on weight and standing height. Waist circumference was measured with a fiberglass measuring tape (Lafayette Instruments Company Inc., USA) at the umbilicus level with subjects in the supine position .
Body composition was measured by bioimpedance analysis (BIA) (Model BIA-103, RJL System, Detroit, MI, USA). Volunteers were instructed to lie supine, with their hands at their sides and their legs separated. The skin surface was cleaned with ethanol, and the electrodes were placed on the dorsum of the right foot and hand [28, 29]. The model used to predict fat-free mass, and percentage of body fat was based on an algorithm validated for Mexican adults: , where: is height in cm, is resistance in ohms, is body weight in kg, age is in years, and is reactance in ohms .
A breath test was performed on the same day as the other measurements. A baseline breath sample was taken, then subjects were instructed to ingest 100 mL of tap water containing of 25 mg universally labeled [13C]glucose (with all six carbons 13C) (Market Biosciences Corporation, Columbia, MD) and 15 g of USP-Dextrose, as reported in previous studies . Postdose breath samples were obtained at 30, 60, and 90 minutes. To collect breath samples, the subjects were asked to expire normally through a straw into 10 mL gas Exetainer tubes (Labco Exetainer System, 13C and gas testing vials, Labco Limited, UK). Breath samples were analyzed by isotope ratio mass spectrometer (IRMS) (Finnigan, Breath MAT Plus, Bremen, Germany). Basal and 90-minute postdose breath samples were used to compare the absolute increase in 13C, expressed as δ in ‰ [18, 30–32].
3. Measurement of Total CO2 Production
In parallel to the breath test, we also measured total oxygen consumption (VO2), total carbon dioxide production (VCO2), metabolic rate (RMR), and respiratory quotient (RQ) using a ventilated hood indirect calorimetry unit (Deltatrac II; Sensor Medics, USA) [22, 33]. The Deltatrac gas analyzer was calibrated for each run with a span gas mixture of 95.9% O2 and 4.1% CO2. Atmospheric pressure was calibrated using an independent aneroid barometer (Cole Parmer, USA). Room temperature was kept between 24 and 26°C and relative humidity 50–60%. The Deltatrac system was regularly checked by burning known amounts of propane gas at rates similar to the subjects’ energy expenditure within their body weight range. Gas burn recoveries outside the range of ±2.0% O2 and CO2 were used to adjust respiratory gas exchange of individual measurements.
Under fasting conditions, all subjects rested for at least 30 minutes at the metabolic unit, then underwent the measurement protocol. Each measurement was preceded by a 10-minute acclimatization period under the canopy; data from this period were not included in the analysis. Basal breath samples were obtained before giving the breath test solution. Deltatrac measurement was restarted, and VCO2 was permanently recorded in mL/min. Measurements were also stopped at 30, 60, and 90 minutes after the consumption of the solution to collect breath samples.
4. Statistical Analysis
Data are expressed as means and standard deviation. Normality was verified, and log transformation was applied when necessary and expressed as geometric means and 95% confidence intervals. A Pearson’s correlation matrix was used to assess the correlation among variables; in particular, the associations between the increase of expired 13CO2 at 90 minutes after dose with HOMA and fasting insulin were analyzed before and after adjusting for the total CO2 production derived from indirect calorimetric measurements. The diagnostic performance of the 13CO2 method and the accuracy of the test to differentiate between diseased and normal cases was evaluated using receiver operating characteristic (ROC) curve analysis [34, 35]. The ROC curve was also used to determine the point of insulin resistance. The selected cut-off values corresponded to the highest sensitivity and specificity. Positive and negative predictor values and their 95% confidence intervals (95% CI) were obtained in the analysis, as well as the area under the curve. Using the cut-off point to define insulin resistance by 13CO2, participants were divided into insulin resistance individuals and non-insulin-resistant individuals. Clinical and metabolic variables were checked for normality and log transformed if required. Differences between these groups were analyzed using Student’s t-test using a general statistical software (Number Cruncher Statistical System for Windows, Version 2007 Kaysville, USA). Receiver operating characteristic curves were constructed and areas under the curve (AUC) were estimated with their corresponding 95% CI by using the statistical software MedCalc 2009 (Medical Calculator, Mariakerke, Belgium). Comparison of correlation coefficients was done using Fischer’s transformation to set confidence limits.
The study included 58 adult subjects (38 female and 20 male) whose clinical and metabolic characteristics are presented in Table 1.
Figure 1 shows an inverse correlation between HOMA as a marker of insulin resistance and breath test δ13CO2 (‰), with a correlation coefficient of −0.41 () at 90 minutes after dose. Total CO2 production is also related to body size, and the sample included a wide range of BMI’s (18.5 to 48.7); therefore, over the same 90-minute period, total CO2 production was also measured by a ventilated hood calorimetric method, and this data was used to correct the association between HOMA and breath test results. Table 2 shows the correlations between δ13CO2 (‰) breath test results and variables related to insulin resistance, before and after adjusting for total CO2 production during the test. After adjusting for total CO2 production, δ13CO2 correlations with HOMA and fasting glucose were less strong but remained significant; however, correlations with fasting insulin, waist circumference, BMI, and percent body fat were no longer significant ().
A ROC curve was constructed using δ13CO2 (‰) and HOMA values, with sensitivity and specificity calculated at different levels. Figure 2 shows the predictor ROC curve; the y-axis shows the sensitivity and the x-axis corresponds to the false positive rate of the breath test. The highest values of sensitivity and specificity were used to determine the “cut-off point” of 9.99‰ δ13CO2, which corresponded to a sensitivity of 80.0 (95% CI: 51.9, 95.7) and a specificity of 67.4 (95% CI: 51.5, 80.9). Individuals below this cut-off point of 9.99‰ δ13CO2 were considered insulin resistant.
Of the 58 subjects evaluated, 15 were classified as insulin resistant by the breath test δ13CO2 data; 12 of these 15 were also deemed IR by HOMA results (true positives, TP), whilst 3 were not IR (false negatives, FN). Out of the 58 subjects, 43 were not classified as IR by the breath test δ13CO2 data and of these results, 14 were false positive and 29 were also classified as being insulin sensitive by HOMA (true negatives, TN).
Clinical and metabolic variables were determined to be associated with insulin resistance or insulin sensitivity based on the δ13CO2 from the breath test, with a cut-off point of 9.99‰. Waist circumference and BMI were higher in individuals with IR by the breath test (), while percent body fat was not (). HbA1c, triglycerides, and systolic and diastolic blood pressure were higher in the IR group ( and , resp.), while HDL cholesterol was lower () and LDL differences by group did not reach significance.
This research focused on improving the estimation of insulin resistance using a previously proposed noninvasive method based on a [13C]glucose breath test . In normal individuals, the presence of insulin causes glucose uptake to occur in a variety of cells, after which glucose either undergoes glycolysis and oxidation or is shunted to fat synthesis. In either case, CO2 is produced as a metabolic by product that enters into circulation and is eliminated by the lungs. The general assumption behind the breath test method is that ingested [13C]glucose will result in the expiration of detectable 13CO2. In cases of diabetes, glucose intolerance, and different stages of insulin resistance, glucose uptake would be impaired resulting in reduced rate of generation of 13CO2 .
As expected, our data showed the smallest 13CO2 values ( ‰) in the group of diabetic subjects, and the [13C]glucose breath test results showed a significant inverse association with HOMA (; ). Lewanczuk et al.  initially compared the results of the breath test to the gold standard (the euglycemic clamp) and other surrogates methods, such as QUICKI and HOMA, in 26 subjects and found a higher correlation with the euglycemic clamp (), than with HOMA (). In 2009, Banerjee et al.  found a significant correlation between results of HOMA and the breath test () in 98 subjects, which was similar to the correlation observed in the present study. Jetha et al.  also evaluated the [13C]glucose breath test in a Caucasian pediatric prepubertal sample and found a significant association between the breath test and the surrogate HOMA method (, ) [20, 21]. All of these studies showed a clear relationship between the [13C]glucose breath test results and different IR markers, suggesting that the proposed method may be an easy and noninvasive option to diagnose IR in at-risk individuals. However, other factors must be considered.
Dillon et al.  presented data from [13C]glucose derived 13CO2 in breath collected during a standard oral glucose tolerance test (OGTT) that was extended to a 10 h sampling period. These data showed a good separation in the appearance of 13CO2 in breath during the standard 1–3 hour oral glucose tolerance test (OGTT), averaging about 7‰ 13CO2 PDB units compared to subjects with a normal OGTT. After 4-5 hours the difference disappears. Our protocol for collecting breath samples was within this period, specifically, at 90 minutes after dose from which the AUC and cut-off points were calculated.
Rating and Langhans  pointed out that one important limitation of the 13C breath test could be endogenous CO2 production, which is influenced by body mass, specifically by the amount of metabolically active cells, as well by the type and quantity of macronutrients in the diet. Elwyn et al.  described differences between oxygen consumption and CO2 production in obese and normal individuals. In our study, we measured total endogenous CO2 production in parallel to the breath test. The total expired volume of CO2 at 90 minutes was used to adjust the correlation between results of the glucose breath test and HOMA. This adjustment changed the value from −0.41 to −0.26; however, these correlation coefficients were not significantly different. Our findings agree with the observations of Rating and Langhans  showing that the δ13CO2 (‰) results depended on endogenous CO2 production [22, 37]. According to these authors, any breath test must take endogenous CO2 production into account for the calculation of actual or cumulative 13CO2 elimination or the 13CO2 peak production rate. It is possible that the correlations found in other studies might be overestimating the association between δ13CO2 (‰) and HOMA or other IR reference methods. No other papers using the [13C]glucose breath test have also measured endogenous CO2.
Sensitivity and specificity analyses were used to determine a cut-off point for the δ13CO2 values (≤9.99‰ 13CO2 for IR individuals). A 2004 patent of Yatscoff et al.  also uses a similar cut-off point of ≤9.0‰ 13CO2; it is important to mention that the comparative methods differed between these studies. In the present study, the test accuracy was determined by the values of sensitivity (80.0) and specificity (67.4); the method mentions an observed value of sensitivity and specificity of 67, similar to our specificity value (67.4) but different from our sensitivity value of 80.
An ideal diagnostic method should have sensitivity and specificity each close to 100, and methods with values lower than 80 should not be considered ; however, is not common to find diagnostic methods with values above 90. In the case of ROC curves, the area under the curve (AUC) is a global measure of accuracy in a diagnostic test  and by itself shows the validity of the test; we found an AUC of 0.74. Morgan et al. and Swets [40, 41] stated that values between 0.5 and 0.7 indicate poor accuracy, while values between 0.7 and 0.9 suggest that the method can be used to reach certain objectives. It should be noted that for clinical applications, higher AUC values might be better indicators than high values of sensitivity and specificity together, because AUC is measure of the test accuracy; additionally AUC allows estimation of the effect size indices, which are directly related to clinical significance, particularly when predictor variables are binary. In clinical settings, screening tests with high sensitivity and relatively low specificity are more valuable when the purpose is to rapidly detect initial phases of disease. IR is a subclinical condition that precedes serious alterations of glucose metabolism; therefore, an appropriate screening strategy should prioritize early detection of IR that may delay or prevent excessive burden of type 2 diabetes.
The [13C]glucose breath test is a simple, safe, and non-invasive method that could be used in a wide variety of individuals, including children and pregnant women. The method itself does not appear to be sufficiently robust for an accurate diagnosis of insulin resistance; however, our findings suggest that the test can be helpful to identify individuals who are not insulin resistant. This screening may contribute to allowing preventive efforts to be better addressed for the development of type 2 diabetes in clinical settings.
No potential conflicts of interest relevant to this article were reported. M. E. Valencia designed the study, oversaw the conduct of the study, wrote the paper, and is the guarantor. E. Ibara-Pastrana recruited subjects, conducted physiological measurements and assisted with writing the manuscript. M. C. C. Plata conducted clinical and biochemical evaluations and contributed to writing the paper. G. Alvarez carried out the statistical analysis and contributed to writing the paper. The authors thank the patients who volunteered to participate in this study. We are also thankful to Christine Slater for her suggestion to consider total CO2 production and they thank A. C. Gallegos from CIAD, A. C. for analysis of breath samples.
World Health Organization Western Health Organization, IASO International Association for the Study of Obesity, International Obesity TaskForce, The Asia-Pacific Perspective: Redefining Obesity and Its Treatment, Health Communications, Sydney, Australia, 2000.
S. Mitall, The Metabolic Syndrome in Clinical Practice, Springer, London, UK, 2007.
G. M. Reaven, “Pathophysiology of insulin resistance in human disease,” Physiological Reviews, vol. 75, no. 3, pp. 473–486, 1995.View at: Google Scholar
W. T. Cefalu, “Insulin resistance: cellular and clinical concepts,” Proceedings of the Society for Experimental Biology and Medicine, vol. 226, no. 1, pp. 13–26, 2001.View at: Google Scholar
L. C. Groop, “Insulin resistance: the fundamental trigger of type 2 diabetes,” Diabetes, Obesity and Metabolism, vol. 1, no. 1, pp. 1–7, 1999.View at: Google Scholar
E. Ferrannini, “Insulin resistance is central to the burden of diabetes,” Diabetes/Metabolism Reviews, vol. 13, pp. 81–86, 1998.View at: Google Scholar
“Prevention or delay of type 2 diabetes,” Diabetes Care, vol. 27, p. s47, 2004.View at: Google Scholar
The Diabetes Prevention Program Research Group, “Within-trial cost-effectiveness of lifestyle intervention or metformin for the primary prevention of type 2 diabetes,” Diabetes Care, vol. 26, pp. 2518–2523, 2003.View at: Google Scholar
R. A. DeFronzo, J. D. Tobin, and R. Andres, “Glucose clamp technique: a method for quantifying insulin secretion and resistance,” The American Journal of Physiology, vol. 237, no. 3, pp. E214–E223, 1979.View at: Google Scholar
A. Katsuki, Y. Sumida, E. C. Gabazza et al., “Homeostasis model assessment is a reliable indicator of insulin resistance during follow-up of patients with type 2 diabetes,” Diabetes Care, vol. 24, no. 2, pp. 362–365, 2001.View at: Google Scholar
D. R. Matthews, J. P. Hosker, and A. S. Rudenski, “Homeostasis model assessment: insulin resistance and β-cell function from fasting plasma glucose and insulin concentrations in man,” Diabetologia, vol. 28, no. 7, pp. 412–419, 1985.View at: Google Scholar
S. Melmed and P. Michael Conn, “Insulin secretion and action,” in Endocrinology: Basic and Clinical Principles, H. Press, Ed., pp. 311–319, Human Press, Totowa, NJ, USA, 2nd edition, 2005.View at: Google Scholar
M. M. Jetha, U. Nzekwu, R. Z. Lewanczuk, and G. D. C. Ball, “A novel, non-invasive 13C-glucose breath test to estimate insulin resistance in obese prepubertal children,” Journal of Pediatric Endocrinology and Metabolism, vol. 22, no. 11, pp. 1051–1059, 2009.View at: Google Scholar
D. Banerjee, N. Vikram, P. Mishra, R. Bhatt, S. Prakash, and A. Misra, “Correlation of a [13C] glucose breath test with surrogate markers of insulin resistance in Urban and Rural Asian Indians,” Metabolic Syndrome and Related Disorders, vol. 7, no. 3, pp. 215–220, 2009.View at: Publisher Site | Google Scholar
H. E. Elwyn, J. Askanazi, J. M. Kinney, and S. Bursztein, Energy Metabolism, Indirect Calorimetry, and Nutrition, William & Wilkins, Baltimore, Md, USA, 1989.
A. B. Saraí, “Fórmulas para el cálculo de la muestra en investigaciones de salud,” Salud Tab, vol. 11, pp. 333–338, 2005.View at: Google Scholar
Organization WHO, “Definition and diagnosis of diabetes mellitus and intermediate hyperglycemia :report of a WHO/IDF consultation,” Tech. Rep., WHO, Geneva, Switzerland, 2006.View at: Google Scholar
D. Perloff, C. Grim, J. Flack et al., “Human blood pressure: determination by sphygmomanometry,” Circulation, vol. 88, no. 5 I, pp. 2460–2470, 1993.View at: Google Scholar
R. S. Gibson, Principles of Nutritional Assessment, Oxford University, Oxford, NY, USA, 2nd edition, 2005.
H. C. Lukaski, P. E. Johnson, W. W. Bolonchuk, and G. I. Lykken, “Assessment of fat-free mass using bioelectrical impedance measurements of the human body,” American Journal of Clinical Nutrition, vol. 41, no. 4, pp. 810–817, 1985.View at: Google Scholar
H. Lukaski, “Evaluation of body composition: why and how?” Mediterranean Journal of Nutrition and Metabolism, vol. 2, pp. 1–10, 2009.View at: Google Scholar
V. Iyengar, “Nuclear and isotopic techniques for addressing nutritional problems, with special reference to current applications in developing countries,” Food and Nutrition Bulletin, vol. 23, no. 1, pp. 3–10, 2002.View at: Google Scholar
OIdE. Atómica, Mejorando la Nutrición a Través de la Ciencia Nuclear, Organismo Internacional de Energia Atomica, Vienna, Austria, 2004.
A. Barrado, T. Preston, C. Slater et al., “The usefulness of stable isotopes in nutrition and human health: the application of mass spectrometry and 13C-breath tests to detect helicobacter pylori infection,” Archivos Latinoamericanos de Nutricion, vol. 54, no. 4, pp. 27–43, 2004.View at: Google Scholar
H. Alemán-Mateo, G. Salazar, M. Hernández-Triana, and M. E. Valencia, “Total energy expenditure, resting metabolic rate and physical activity level in free-living rural elderly men and women from Cuba, Chile and México,” European Journal of Clinical Nutrition, vol. 60, no. 11, pp. 1258–1265, 2006.View at: Publisher Site | Google Scholar
C. E. Metz, B. A. Herman, and J.-H. Shen, “Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data,” Statistics in Medicine, vol. 17, pp. 1033–1053, 1998.View at: Google Scholar
M. H. Zweig and G. Campbell, “Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine,” Clinical Chemistry, vol. 39, no. 4, pp. 561–577, 1993.View at: Google Scholar
D. Rating and C. D. Langhans, “Breath tests: concepts, applications and limitations,” European Journal of Pediatrics, vol. 156, no. 1, pp. S18–S23, 1997.View at: Google Scholar
R. W. Yatscoff , R. T. Foster, L. J. Aspeslet, and R. Lewanczuk, “13C glucose breath test for the diagnosis of diabetic indications and monitoring glycemic control,” Isotechnika, USA, 2001.View at: Google Scholar
M. J. Bargueño, J. L. García-Bastos, and J. M. González-Buitrago, “Las Curvas ROC en la Evaluación de las Pruebas Diagnósticas,” Medicina Clínica, vol. 104, pp. 661–670, 1995.View at: Google Scholar
G. A. Morgan, J. A. Gliner, and R. J. Harmon, Understanding and Evaluating Research in Applied and Clinical Settings, Lawrence Erlbaum, Mahwah, NJ, USA, 2006.
J. A. Swets, “Measuring the accuracy of diagnostic systems,” Science, vol. 240, no. 4857, pp. 1285–1293, 1988.View at: Google Scholar