Inflammatory Biomarkers in CancerView this Special Issue
Research Article | Open Access
Monika Zajkowska, Edyta Katarzyna Głażewska, Grażyna Ewa Będkowska, Przemysław Chorąży, Maciej Szmitkowski, Sławomir Ławicki, "Diagnostic Power of Vascular Endothelial Growth Factor and Macrophage Colony-Stimulating Factor in Breast Cancer Patients Based on ROC Analysis", Mediators of Inflammation, vol. 2016, Article ID 5962946, 8 pages, 2016. https://doi.org/10.1155/2016/5962946
Diagnostic Power of Vascular Endothelial Growth Factor and Macrophage Colony-Stimulating Factor in Breast Cancer Patients Based on ROC Analysis
Breast cancer (BC) is the most common malignancy in women. Vascular endothelial growth factor (VEGF) has been described as an important regulator of angiogenesis which plays a vital role in the progression of tumor. Macrophage colony-stimulating factor (M-CSF) is a cytokine whose functions include regulation of hematopoietic lineages cells growth, proliferation, and differentiation. We investigated the diagnostic significance of these parameters in comparison to CA15-3 in BC patients and in relation to the control group (benign breast tumor and healthy women). Plasma levels of the tested parameters were determined by ELISA and CA15-3 was determined by CMIA. VEGF was shown to be comparable to CA15-3 values of sensitivity in BC group and, what is more important, higher values in early stages of BC. VEGF was also the only parameter which has statistically significant AUC in all stages of cancer. M-CSF has been shown to be comparable to CA15-3 and VEGF, specificity, and AUC values only in stages III and IV of BC. These results indicate the usefulness and high diagnostic power of VEGF in the detection of BC. Also, it occurred to be the best candidate for cancer diagnostics in stages I and II of BC and in the differentiation between BC and benign cases.
Breast cancer (BC) is an important health problem worldwide. Each year the incidence rate of this disease increases significantly. In 2015, only in the United States about 231,840 women were diagnosed with BC and 40,290 of them died . This disease may appear at any age, yet a particularly high risk is related to females after 50 years of age, what is correlated with menopausal hormonal changes .
The crucial factor influencing a successful treatment and survival rate of BC patients is early diagnosis. Biochemical detection of this tumor is nowadays based on markers such as CA 15-3, CEA, and CA 27.29 . In view of their insufficient specificity and sensitivity at the initial type of BC, scientists around the world perform intensive research to find better biomarkers whose levels would correlate with the presence and stage of the studied disease. We assumed that these factors may be cytokines: vascular endothelial growth factor (VEGF) and macrophage colony-stimulating factor (M-CSF).
VEGF has been described as an important regulator of angiogenesis, a crucial process of tumor invasion and progression . Significantly increased levels of VEGF have been found in the serum or plasma of patients suffering from breast and gynecological tumors, for example, ovarian or cervical, as well as other kinds of cancers . The in vitro and in vivo studies performed so far presented that the overexpression of this cytokine leads to cancer growth and metastasis, while the inhibition of VEGF resulted in the suppression of tumor development .
In contrast, M-CSF is a cytokine whose functions include regulation of hematopoietic lineages cells growth, proliferation, and differentiation . M-CSF is produced pathologically by cancer cells. The overexpression of this cytokine has been detected in a variety of tumors, female reproductive tract cancers and breast, renal, colorectal, pancreatic, prostate, and head and neck tumors, and has been correlated with poor prognosis [8, 9]. What is interesting, circulating level of M-CSF has been found to be useful as a method of estimating patients’ survival rates.
As VEGF and M-CSF play a significant role in carcinogenesis, the aim of the present study was to investigate the diagnostic power of the selected cytokines and a comparative marker CA 15-3 in breast tumor detection.
In this paper, the use of healthy volunteers and women with benign breast lesions together as a one control group better reflects the current population of women. The data obtained in this work may prove the usefulness of the analyzed parameters (separately and together) in the detection of BC, as a new diagnostic panel.
2. Materials and Methods
Table 1 shows the tested groups. The study included 120 breast cancer (BC) women diagnosed by the oncology group. The breast cancer patients were treated in the Department of Oncology, Medical University of Bialystok, Bialystok, Poland. Tumor classification and staging were done in accordance with the International Union Against Cancer Tumor-Node-Metastasis (UICC-TNM) classification. The breast cancer histopathology was established in all cases by tissue biopsy of mammary tumor or after surgery from tumor cancer tissues (all patients with ductal adenocarcinoma). The pretreatment staging procedures included physical and blood examinations, mammography, mammary ultrasound scanning, breast core biopsies, and chest X-rays.
In addition, radioisotopic bone scans, examination of bone marrow aspirates, and CT scans of brain and chest were performed when necessary. None of the patients had received chemotherapy or radiotherapy before blood sample collection.
The control group included 120 patients (60 patients with benign breast tumor, adenoma, intraductal papilloma, fibroadenoma, mastopatia, and 60 healthy untreated women) who underwent mammary gland examination performed by a gynecologist prior to blood sample collection. In addition, mammary ultrasound scanning was performed in all cases. The benign breast tumor histopathology was established in all cases by tissue biopsy of mammary tumor or after surgery.
The study was approved by the local Ethics Committee in Medical University of Bialystok (R-I-002/239/2014). All the patients gave their informed consent for the examination.
2.2. Biochemical Analyses
Venous blood samples were collected from each patient into a heparin sodium tube, centrifuged at 1000 rpm for 15 min to obtain plasma samples and stored at –85°C until assayed. The tested parameters were measured with the enzyme-linked immunosorbent assay (ELISA) (VEGF and M-SCF, Quantikine Human Immunoassay, R&D Systems Inc., Minneapolis, MN, USA) and chemiluminescent microparticle immunoassay (CMIA) (CA 15-3, Abbott, Chicago, IL, USA). According to the manufacturer’s protocols, duplicate samples were assessed for each standard, control, and sample.
The intra-assay coefficient of variation (CV%) of CA 15-3 is reported to be 2.2% at a mean concentration of 27.0 U/mL (SD = 0.6). VEGF is reported to be 4.5% at a mean concentration of 235 pg/mL (SD = 10.6). M-CSF is reported to be 3.4% at a mean concentration of 227 pg/mL (SD = 7.7).
The interassay coefficient of variation (CV%) of CA 15-3 is reported to be 2.6% at a mean concentration of 27.0 U/mL (SD = 0.7). VEGF is reported to be 7.0% at a mean concentration of 250 pg/mL (SD = 17.4). M-CSF is reported to be 3.1% at a mean concentration of 232 pg/mL (SD = 7.3).
2.3. Statistical Analysis
In this analysis we have used healthy volunteers and women with benign breast lesions together as a one control group. This is in accordance with the latest published papers especially for ROC analysis [10–13]. Statistical analysis was performed by using STATISTICA 12.0. We have defined the receiver-operating characteristics (ROC) curve for all the tested parameters and CA 15-3. The construction of the ROC curves was performed using GraphROC program for Windows and the areas under ROC curve (AUCs) were calculated to evaluate the diagnostic accuracy and to compare AUCs for all tested parameters separately and in combination with a commonly used tumor marker (CA 15-3). Statistically significant differences were defined as comparisons resulting in .
The cut-off values were calculated by Youden’s index (as a criterion for selecting the optimum cut-off point) and for each of the tested parameters they were as follows: VEGF, 70.25 pg/mL; M-CSF, 394.38 pg/mL; and CA 15-3, 18.30 U/mL.
Table 2 shows the sensitivity (SE) and specificity (SP) of the investigated parameters and CA 15-3. We indicated that the SE of the tested parameters in the total cancer group was the highest for CA 15-3 (83.75%) and slightly higher than that for VEGF (76.25%) and M-CSF (60%). Among all parameters, the highest SE in stage I of cancer was observed for VEGF (75%), in stage II of BC it was observed for VEGF and CA 15-3 (75%, equal for both parameters), and in stages III and IV of BC it was observed for CA 15-3 (95% and 100%, resp.).
The diagnostic SP of the tested parameters was the highest for M-CSF and VEGF (90% and 85%, resp.) and was higher than that for CA 15-3 (75%).
The combined analysis for VEGF or M-SCF with CA 15-3 in the total group of BC resulted in a high increase in SE in both cases (96.25% and 91.25%, resp.). A similar range in the total BC group was obtained for the combination of VEGF, M-SCF, and CA 15-3 (96.25%). In all combinations, SP dropped slightly in comparison to the analysis of single parameters.
The relationship between the diagnostic SE and SP is illustrated by the ROC curve. The area under the ROC curve (AUC) indicates the clinical usefulness of a tumor marker and its diagnostic power. It also quantifies the overall ability of the test to differentiate between the individuals with the disease and those without it. All data related to AUCs in different stages of BC (I–IV) are included in Table 3.
|, statistically significantly larger AUCs compared to AUC = 0.5.|
We noticed that the VEGF area under the ROC curve (0.729) in the total group of breast cancer was higher than the area of CA 15-3 (0.698) and M-CSF (0.645), statistically significantly larger in comparison to AUC = 0.5, borderline of the diagnostic usefulness of the test ( in all cases). The combined analysis of VEGF or M-SCF with CA 15-3 in the total group of BC resulted in a slight increase in AUCs in both cases (0.753 and 0.699, resp.), but a maximum range in the total BC group was obtained for the combination of VEGF, M-SCF, and CA 15-3 (0.754) ( in all cases) (Figure 1).
In stage I of BC the highest AUC of all the tested parameters was found in VEGF (0.691) and it was the only parameter which was statistically significantly larger in comparison to AUC = 0.5 () (Figure 2).
In stage II of BC the highest AUC of all tested parameters was also observed in VEGF (0.716; ). The combined analysis of VEGF with CA 15-3 (0.629; ) and combination of all tested parameters showed a slight decrease in AUC (0.629; ) (Figure 3).
In stage III of BC the highest AUC of all the tested parameters was observed in CA 15-3 (0.819; ) and it was slightly higher than VEGF (0.818; ) and M-CSF (0.811; ). The combined analysis of VEGF or M-CSF with CA 15-3 showed an increase in AUC values (0.878 and 0.850, resp.) ( in both cases), but the maximum range in stage III of BC was obtained for the combination of VEGF, M-SCF, and CA 15-3 (0.879; ) (Figure 4).
In stage IV of BC the highest AUC of all the tested parameters was found in CA 15-3 (0.893; ) and it was higher than M-CSF (0.834; ) and VEGF (0.690; ). The combined analysis of VEGF with CA 15-3 or all tested parameters showed an increase in AUC values (0.908; in both cases), but the maximum range in stage IV of BC was obtained for the combination of M-SCF and CA 15-3 (0.921; ) (Figure 5).
Angiogenesis is a vital blood vessel formation process in tumor progression and nutrition. VEGF is considered to be an important factor in promoting angiogenesis and cell proliferation in many pathological conditions. High levels of VEGF have been found in different kinds of tumors, for example, gastric  or colorectal cancer , and also in gynecological malignancies such as ovarian  or cervical cancer . High plasma levels of VEGF have been also found in breast cancer .
Tumor growth is influenced by a variety of external and internal factors. Our immune system (producing growth factors and cytokines) is one of the most important mediators involved in tumor development. M-CSF belongs to the group of hematopoietic growth factors (HGFs) which are overexpressed in many tumors. The main function of M-CSF is regulation and differentiation of hematopoietic progenitor cell growth. Its high levels have been found in gastric  and pancreatic cancer . It has also been found in many types of gynecological malignancies, for example, ovarian [20, 21], cervical , or endometrial cancer , and it has also been found in breast cancer .
Sensitivity (SE) measures the proportion of positives that are correctly identified. In this study, the SE for CA 15-3 was the highest in the total group of breast cancer patients. However, in stages I and II of cancer it was the highest for VEGF which is much more important because such a high sensitivity (75%) allows us to confirm the occurrence of breast cancer in the earliest stages, while contributing to an increase in cancer detection, the course of which is often asymptomatic. Earlier diagnosis is associated with a greater chance of survival as well as quality and length of life of patients with BC. Similar data were observed in our previous studies [5, 8], where CA 15-3 had also the highest values in the total group, but what is more important is the fact that VEGF had a higher value in stage I of BC. However, in opposition to this paper, statistical analysis of those previous publications was conducted on groups of “breast cancer patients versus healthy women” only.
Other researchers, such as Motawa El Husseini et al. , have also indicated very high SE (83.93%) and SP (96.67%) for VEGF in BC diagnostics, but they conducted their study on 51 BC patients and only 30 healthy volunteers as a control group.
We have also observed similar data in other types of cancer, for example, in ovarian cancer . Other researchers, for instance, Kozłowski et al.  in esophageal cancer (SE, 83%; SP, 70%) or Cao et al.  in lung cancer (SE, 81.8%; SP, 84.2%), have obtained similar results for VEGF.
The AUC represents the overall accuracy of a test, with a value approaching 1.0 indicating perfect SE and SP. According to this study, the ROC area of VEGF was the largest of all the tested parameters (even higher than CA 15-3 which is nowadays commonly used in the diagnosis of BC) and is the only parameter for which AUC was statistically significantly larger in comparison to AUC = 0.5 in all stages of BC (I–IV), not only in the total group. This is very important as it indicates higher usefulness of VEGF compared to CA 15-3 in the differentiation between BC and benign breast tumor.
Our results showed that the diagnostic power (AUC) of the tested parameters, especially VEGF, in the total group of BC patients was slightly lower than the one obtained by Zhang et al.  (0.788). The discrepancy between our research and that study may be related to a different number of patients involved in those studies. Other researchers such as Motawa El Husseini et al.  have obtained a higher AUC value for VEGF (0.938), but the control group in their study comprised only healthy women.
The diagnostic power of VEGF in the course of other tumors, for example, studies conducted in lung cancer by Cao et al. , revealed a slightly higher AUC value (0.855) than our outcome, which may be associated with different types of cancer. Other researchers, for example, Kozłowski et al. , have obtained slightly higher results for VEGF (0.865) in esophageal cancer. This may result from the fact that they conducted their study on 30 healthy volunteers in control group (without benign cancer patients). High importance of VEGF in those types of tumors points out that this cytokine seems to be a good biomarker for a variety of cancers, as shown by other researchers. In stage I of BC the highest AUC of all tested parameters was observed for VEGF. In our previous study in BC , which comprised BC patients and only healthy women as a control group, the highest AUC value was found for CA 15-3 (0.7068) and it was the only parameter for which AUC was statistically significantly larger in comparison to AUC = 0.5 (). Present statistical analysis with new, combined control group revealed even better results for tested cytokine (VEGF is a better marker than CA 15-3), which is additionally in opposition to the previous results obtained for M-CSF. In our other study  conducted in ovarian cancer, which compared M-CSF to HE4 and CA 125, the AUC value in stage I was 0.7676 () and was significantly higher than that in this study.
In stage II of BC, only VEGF and the combined analysis of VEGF and CA 15-3 had a statistically significantly larger AUC in comparison to AUC = 0.5. In our previous study in BC  all the tested parameters had significant values (which might be related with the composition of the control group, only healthy subjects). In the study on ovarian cancer  the value of AUC for M-CSF was higher (similarly to stage I) than that in this study, but the control group in this study also comprised only healthy women.
In stages III and IV of BC, all the tested parameters had statistically significantly larger AUC in comparison to AUC = 0.5. In our previous study in BC  all the tested parameters also showed significant values similarly to the study conducted in ovarian cancer .
The combined analysis of VEGF or M-CSF with CA 15-3 resulted in an increase in SE and AUC values, which may be useful in the future diagnosis of this cancer. This study is also similar to our previous paper, indicating diagnostic usefulness of this biomarkers panel in cancer diagnostics. Better parameters were obtained in the combination of VEGF than M-CSF and CA 15-3. The combination of all three parameters did not affect the significant increase in SE, SP, or AUC, which may lead to the assumption that the combination of VEGF and CA 15-3 may be the best diagnostic panel in the diagnosis of BC.
In conclusion, our present results indicate the usefulness and a high diagnostic power of VEGF in the detection of breast cancer. Among the tested parameters, VEGF occurred to be the best candidate for cancer diagnostics (better than commonly used tumor marker, CA 15-3) especially in stages I and II of BC as well as in the differentiation between BC and benign breast tumor. M-CSF has shown low SE in stages I and II and was comparable to CA 15-3 and VEGF, SE, and AUC values in stages III and IV of BC. VEGF, especially in the combination with CA 15-3, showed the highest usefulness and diagnostic power in the detection of breast cancer and may indicate a new panel of biomarkers used in early diagnosis of BC.
This work was conducted in accordance with the Declaration of Helsinki (1964).
The authors declare that there are no competing interests regarding the publication of this paper.
This research was financed by Grants for Medical University of Bialystok (nos. 143-07773F, 143-07774F, and N/ST/MN/16/004/2207) in the years 2014–2016 from the Polish Ministry of Science and Higher Education.
- R. L. Siegel, K. D. Miller, and A. Jemal, “Cancer statistics, 2015,” CA: A Cancer Journal for Clinicians, vol. 65, no. 1, pp. 5–29, 2015.
- D. Trichopoulos, H.-O. Adami, A. Ekbom, C.-C. Hsieh, and P. Lagiou, “Early life events and conditions and breast cancer risk: from epidemiology to etiology,” International Journal of Cancer, vol. 122, no. 3, pp. 481–485, 2008.
- L. Harris, H. Fritsche, R. Mennel et al., “American society of clinical oncology 2007 update of recommendations for the use of tumor markers in breast cancer,” Journal of Clinical Oncology, vol. 25, no. 33, pp. 5287–5312, 2007.
- D. J. Hicklin and L. M. Ellis, “Role of the vascular endothelial growth factor pathway in tumor growth and angiogenesis,” Journal of Clinical Oncology, vol. 23, no. 5, pp. 1011–1027, 2005.
- S. Ławicki, M. Zajkowska, E. Głażewska, G. Będkowska, and M. Szmitkowski, “Plasma levels and diagnostic utility of VEGF, MMP-9, and TIMP-1 in the diagnosis of patients with breast cancer,” OncoTargets and Therapy, vol. 9, pp. 911–919, 2016.
- A. Veeravagu, A. R. Hsu, W. Cai, L. C. Hou, V. C. K. Tse, and X. Chen, “Vascular endothelial growth factor and vascular endothelial growth factor receptor inhibitors as anti-angiogenic agents in cancer therapy,” Recent Patents on Anti-Cancer Drug Discovery, vol. 2, no. 1, pp. 59–71, 2007.
- S. Chockalingam and S. S. Ghosh, “Macrophage colony-stimulating factor and cancer: a review,” Tumor Biology, vol. 35, no. 11, pp. 10635–10644, 2014.
- S. Ławicki, E. K. Głażewska, M. Sobolewska, G. E. Będkowska, and M. Szmitkowski, “Plasma levels and diagnostic utility of macrophage colony-stimulating factor, matrix metalloproteinase-9, and tissue inhibitor of metalloproteinases-1 as new biomarkers of breast cancer,” Annals of Laboratory Medicine, vol. 36, no. 3, pp. 223–229, 2016.
- S. K. Chambers, B. M. Kacinski, C. M. Ivins, and M. L. Carcangiu, “Overexpression of epithelial macrophage colony-stimulating factor (CSF-1) and CSF-1 receptor: a poor prognostic factor in epithelial ovarian cancer, contrasted with a protective effect of stromal CSF-1,” Clinical Cancer Research, vol. 3, no. 6, pp. 999–1007, 1997.
- W. Zhang, X. Zheng, and X. Wang, “Oxidative stress measured by thioredoxin reductase level as potential biomarker for prostate cancer,” American Journal of Cancer Research, vol. 5, no. 9, pp. 2788–2798, 2015.
- Q. Wang, X. Li, S. Ren et al., “Serum levels of the cancer-testis antigen POTEE and its clinical significance in non-small-cell lung cancer,” PLoS ONE, vol. 10, no. 4, Article ID e0122792, 2015.
- Y. Park, J. H. Lee, D. J. Hong, E. Y. Lee, and H. S. Kim, “Diagnostic performances of HE4 and CA125 for the detection of ovarian cancer from patients with various gynecologic and non-gynecologic diseases,” Clinical Biochemistry, vol. 44, no. 10-11, pp. 884–888, 2011.
- D. Cheng, B. Liang, and Y. Li, “Serum Vascular Endothelial Growth Factor (VEGF-C) as a diagnostic and prognostic marker in patients with ovarian cancer,” PLoS ONE, vol. 8, no. 2, Article ID e55309, 2013.
- D. J. Park, N. J. Thomas, C. Yoon, and S. S. Yoon, “Vascular endothelial growth factor a inhibition in gastric cancer,” Gastric Cancer, vol. 18, no. 1, pp. 33–42, 2015.
- T. Goi, T. Nakazawa, Y. Hirono, and A. Yamaguchi, “The anti-tumor effect is enhanced by simultaneously targeting VEGF and PROK1 in colorectal cancer,” Oncotarget, vol. 6, no. 8, pp. 6053–6061, 2015.
- S. Ławicki, G. E. Będkowska, E. Gacuta-Szumarska, and M. Szmitkowski, “The plasma concentration of VEGF, HE4 and CA125 as a new biomarkers panel in different stages and sub-types of epithelial ovarian tumors,” Journal of Ovarian Research, vol. 6, no. 1, article 45, 2013.
- G. E. Będkowska, S. Ławicki, and M. Szmitkowski, “Molecular markers of carcinogenesis in the diagnostics of cervical cancer,” Postępy Higieny i Medycyny Doświadczalnej, vol. 63, pp. 99–105, 2009.
- R. Zhou, Y. Zhou, and Z. Chen, “Exploration of macrophage colony-stimulating factor as a new type of tumor marker,” Biomedical Reports, vol. 1, no. 6, pp. 845–849, 2013.
- G. Vasiliades, N. Kopanakis, M. Vasiloglou et al., “Role of the hematopoietic cytokines SCF, IL-3, GM-CSF and M-CSF in the diagnosis of pancreatic and ampullary cancer,” International Journal of Biological Markers, vol. 27, no. 3, pp. e186–e194, 2012.
- S. Ławicki, E. Gacuta-Szumarska, G. E. Będkowska, and M. Szmitkowski, “Hematopoietic cytokines as tumor markers in gynecological malignancies. A multivariate analysis in epithelial ovarian cancer patients,” Growth Factors, vol. 30, no. 6, pp. 357–366, 2012.
- G. E. Będkowska, S. Ławicki, E. Gacuta, P. Pawłowski, and M. Szmitkowski, “M-CSF in a new biomarker panel with HE4 and CA 125 in the diagnostics of epithelial ovarian cancer patients,” Journal of Ovarian Research, vol. 8, article 27, 2015.
- S. Ławicki, G. E. Będkowska, E. Gacuta-Szumarska, P. Knapp, and M. Szmitkowski, “The plasma levels and diagnostic utility of stem cell factor (SCF) and macrophage-colony stimulating factor (M-CSF) in cervical cancer patients,” Polski Merkuriusz Lekarski, vol. 25, no. 145, pp. 38–42, 2008.
- S. Ławicki, G. E. Będkowska, E. Gacuta-Szumarska, and M. Szmitkowski, “Hematopoietic cytokines as tumor markers in gynecological malignancies: a multivariate analysis with ROC curve in endometrial cancer patients,” Growth Factors, vol. 30, no. 1, pp. 29–36, 2012.
- E. Motawa El Husseini, E. Fatmaelzahraa Hussein, M. Abdelghani, and M. Basma Abdelghany, “Clinical significance of TGF alpha, TGF beta1 and VEGF in Sera of Egyptian patients with breast cancer,” The Egyptian Journal of Hospital Medicine, vol. 52, pp. 555–565, 2013.
- M. Kozłowski, W. Laudański, B. Mroczko, M. Szmitkowski, R. Milewski, and G. Łapuć, “Serum tissue inhibitor of metalloproteinase 1 (TIMP-1) and vascular endothelial growth factor A (VEGF-A) are associated with prognosis in esophageal cancer patients,” Advances in Medical Sciences, vol. 58, no. 2, pp. 227–234, 2013.
- C. Cao, S.-F. Sun, D. Lv, Z.-B. Chen, Q.-L. Ding, and Z.-C. Deng, “Utility of VEGF and sVEGFR-1 in bronchoalveolar lavage fluid for differential diagnosis of primary lung cancer,” Asian Pacific Journal of Cancer Prevention, vol. 14, no. 4, pp. 2443–2446, 2013.
- J. Y. Zhang, Y. Li, J. Z. Wu et al., “Detection of serum VEGF and MMP-9 levels by luminex multiplexed assays in patients with breast infiltrative ductal carcinoma,” Experimental and Therapeutic Medicine, vol. 8, no. 1, pp. 175–180, 2014.
- S. Ławicki, G. E. Będkowska, and M. Szmitkowski, “VEGF, M-CSF and CA 15-3 as a new tumor marker panel in breast malignancies: a multivariate analysis with ROC curve,” Growth Factors, vol. 31, no. 3, pp. 98–105, 2013.
Copyright © 2016 Monika Zajkowska et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.