Abstract
Since 2001 the Istituto Superiore di Sanità established a quality assurance programme for molecular genetic testing that covers four pathologies: Cystic Fibrosis (CF), Beta Thalassemia (BT), Fragile X Syndrome (FX), and Familial Adenomatous Polyposis Coli (APC). Since 2009 this activity is an institutional activity and participation is open to both public and private laboratories. Seven rounds have been performed until now and the eighth is in progress. Laboratories receive 4 DNA samples with mock clinical indications. They analyze the samples using their routine procedures. A panel of assessors review the raw data and the reports; all data are managed through a web utility. In 2010 the number of participants was 43, 17, 15, 5 for CF, BT, FX, APC schemes respectively. Genotyping results were correct in 96%, 98.5%, 100%, and 100% of CF, BT, FX, and APC samples, respectively. Interpretation was correct in 74%, 91%, 88%, and 60% of CF, BT, FX, and APC reports, respectively; however in most of them it was not complete but a referral to genetic counseling was given. Reports were satisfactory in more than 60% of samples in all schemes. This work presents the 2010 results in detail comparing our data with those from other European schemes.
1. Introduction
Since the human genome sequencing was completed, the number of diseases for which genetic tests are available has grown rapidly (2500 diseases for which genetic tests were available in 2011—http://www.ncbi.nlm.nih.gov/projects/GeneTests/static/whatsnew/labdirgrowth.shtml).
Genetic tests are unique in their kind; they are performed only once in the life of a patient because their outcome never changes. Therefore an error may have harmful consequences on the choice of clinical/therapeutic planning and can significantly affect the life choices of the patients and their family.
Laboratories that perform genetic tests are required to work to very high quality standards; monitoring such laboratories is an obligation for the National Health System as part of its mandate to protect the health and quality of life of citizens.
The role of External Quality Assessments (EQA) in ensuring good laboratory practice is recognized at national and international level [1–5]. EQA schemes are the main tools for measuring the quality of laboratory results, for maintaining confidence in molecular genetic tests, and for implementing the standards of quality assurance [6–9]. A number of initiatives were taken internationally to improve quality in genetic testing services, for example, Cystic Fibrosis Quality Network and EMQN in Europe, and CAP in the USA [10, 11].
In 2001 the Italian National Centre for Rare Diseases of the Istituto Superiore di Sanità (ISS, Rome) established EQA schemes for both molecular genetic testing and classical cytogenetic. In particular, for molecular genetic testing, it offers specific EQA schemes for 4 diseases: Cystic Fibrosis (CFTR gene) (CF), Beta Thalassemia (HBB gene) (BT), Fragile X-Syndrome (FMR1 gene) (FX), and Familial Adenomatous Polyposis Coli (APC gene) (APC) [12].
The Italian EQA (IEQA) has primarily an educational role and it aims to improve the quality of genetic tests used in clinical practice [12].
Until now seven rounds have been completed and overall 91 different laboratories have been monitored in the context of the (IEQA). National experts have assessed laboratory performance on genotyping, interpretation, and reporting of test results for a total number of 3158 samples.
A web utility was developed in 2008 to support this activity; it represents a computer interface, among the ISS, the laboratories, and the assessors, that facilitates communication, simplifies data archiving, and minimizes paper usage.
In 2009 (VII round, 2010) the activity was published in the Official Bulletin of the Italian Republic [13]. Participation to IEQA is voluntary and open to both public and private laboratories; laboratories pay a fee to participate.
During the first six rounds the assessment focused in particular on genotyping results and on completeness of reports to evaluate technical ability and harmonize reports among Italian laboratories. Results show that genotyping is in general of good quality whereas reporting presents much larger variations between laboratories and generally a lack of information [14–17]. In the seventh round assessors focused their attention on the ability of laboratories to accurately detect mutations and, in particular, on the ability to interpret the results.
In this work we describe the results of the seventh round of IEQA.
2. Materials and Methods
2.1. Organization of the IEQA
IEQA is organized and coordinated by the National Centre of Rare Diseases of the ISS [12]. Participation is open to both public and private laboratories. Since 2008 the activity is supported by the web utility.
Schemes are strictly anonymous and the identity of laboratories is known only to the ISS. The IEQA scheme organizer and national experts provide advice on the scientific context of the scheme and take decisions and educational actions for the development of the program.
In every scheme ISS provides 4 validated samples of genomic DNA for each round; all samples are distributed with mock data identifications, mock clinical information, and technical data.
Laboratories are asked to test samples using their routine protocols and to provide results of genotyping (raw data) and a full interpretative report in their normal laboratory style by a given deadline: 60 days for CF, BT, and FX; 90 days for APC.
Laboratory results are evaluated by assessors according to established criteria and results are available to laboratories in the reserved area on the web utility.
2.2. Sample Collection and Validation
The genomic DNA samples were obtained from peripheral blood and lymphoblastoid cell lines collected, respectively, by a clinical hospital (Presidio Ospedaliero Microcitemico, Cagliari) (only for the BT scheme) and by biobanks (Galliera Genetic Bank, Genova, and Coriell Cell Repository, Camden, NJ). Two independent working units in the ISS were responsible for DNA samples processing and validation [12].
Mutations carried by samples are validated in the ISS by routine methods: (1) direct sequencing for APC [17]; (2) PCR and Southern Blot for FX [15]; (3) PCR, sequencing and Reverse Dot Blot (RDB) for BT [16]; (4) PCR and RDB Kit according to manufacturing protocols (INNO-LiPA CFTR19 and INNO-LiPA CFTR17+Tn, Innogenetics, Belgium) for CF. Each laboratory received four aliquots of 7 μg, 7 μg, 20 μg, and 40 μg of validated DNA for the CF, BT, APC, and FX scheme, respectively. Table 1 lists samples with mock clinical data and mutations submitted during the VII round.
2.3. Web Utility
The web utility was developed in 2008 and was designed to simplify communications and data sharing among ISS, laboratories, and assessors.
Laboratories receive an identification code (ID) and password (PW) to access the personal area (http://www.iss.it/site/cnmr/privato/cqtg/entry.asp) and the scheme area where they find samples data and instructions to participate in the EQA.
Upon completing the analysis of the samples, the laboratories upload the raw data (jpg format) and the reports (pdf format).
At deadline the data are made available to the assessors who access their reserved area; they assess the results of the laboratories and write their observations in a schedule that is forwarded to the ISS via the web utility.
Final results are uploaded in the reserved area of each laboratory which, however, is informed by e-mail. A report with a summary of all anonymous results is also included in the reserved area and published on the website (http://www.iss.it/cnmr/tege/qual/cont.php?id=90&lang=1&tipo=4).
2.4. Assessment
National experts evaluated the laboratory results twice: first online and then in a meeting at the ISS. Assessment took into account technical performance (raw data), genotyping, interpretation, and reports. All data were treated anonymously and the identity of each laboratory is unknown to the assessors.
Following the assessment, participating laboratories received a feedback with an evaluation of the results. Since 2009 a marking system has been introduced and laboratories receive a mark for genotyping, interpretation, and reporting of results for each sample including comments or suggestions, if necessary. Until now poor performance has neither been assigned nor penalized.
2.5. Evaluation Criteria
Evaluation criteria have been established by the ISS and a panel of national experts taking into account the EMQN criteria, national and international best practice guidelines and publications [18–20].
For each scheme, three assessment topics are identified; a maximum score of 5 points is assigned to each topic for genotyping performance, and of 4 points for interpretation and for reporting. In accordance with the evaluation criteria, points are subtracted for errors and for lack of important information. The presence of raw data and reports and the correct identification of the genotype are necessary preconditions for making the assessment. If the genotyping result is not correct, assessors do not mark the interpretation and the report, but write a comment as feedback to help the laboratories to improve their performance.
The main common elements of the evaluation criteria for all schemes are listed in Table 2; specific topics were discussed by the assessors from time to time within the framework of each scheme.
3. Results
Fifty-six different laboratories participated in the VII round of the IEQA: 41 laboratories affiliated with the public health system and 15 private laboratories. In particular, 43 laboratories participated for CF, 17 for BT, 15 for FX, and 5 for APC. 22 laboratories participated in more than one scheme. During this round 320 samples were analyzed and results were examined by the assessors to evaluate their performance.
3.1. Genotyping Results
3.1.1. Cystic Fibrosis
Overall, 172 samples were sent to the laboratories by the ISS. Genotypes were correctly detected in 165/172 (96%) samples. Genotyping errors occurred in 4% of samples (Table 3). Four laboratories gave no information about the methods used to analyze 14 samples.
3.1.2. Beta Thalassemia
Overall, 68 samples were sent to the laboratories by the ISS. Genotypes were correctly detected in 67/68 (98.5%) of samples. Genotyping errors occurred in 1/68 (1.5%) samples (Table 3).
3.1.3. Fragile X Syndrome
Overall, 60 samples were sent to the laboratories by the ISS. Genotypes were correctly detected in 60/60 samples (100%), but errors occurred in (CGG)n repeat quantification (Table 3). Moreover in one sample the methylation test was not reported.
3.1.4. Familial Adenomatous Polyposis of the Colon (Gene APC)
Overall, 20 samples were sent to the laboratories by the ISS. All samples were correctly genotyped for pathogenic mutations. However one laboratory did not identify an additional gene variant present in one sample (Table 3).
Raw data of 3 samples analyzed by the same laboratory did not have good quality.
3.2. Interpretation Results
Figure 1 shows detailed results on the assessment of genotype interpretation and includes also samples that were not assessed for incorrect genotype; Table 4 shows the information that is most commonly missing for all schemes.
3.2.1. Cystic Fibrosis, Beta Thalassemia, and Fragile X Syndrome
Correct interpretation was reported in 71%, 91%, and 88% of CF, BT, and FX cases, respectively, even though a lack of information was found in the majority of them (i.e., 86%, 64%, and 77% of CF, BT, and XF reports, resp.). It has to be underlined that a referral to genetic counseling was present in most reports not complete (i.e., 87%, 72%, and 80% of CF, BT, and XF reports, resp.).
There was not any interpretation of genotyping results in 25%, 6%, and 3% of CF, BT, and FX reports, respectively; however most CF and FX reports refer to genetic counseling (i.e., 27/43 CF and 2/2 FX reports).
3.2.2. Familial Adenomatous Polyposis of the Colon (Gene APC)
Interpretation was correct in 12/20 (60%) of reports, even though it was not complete in the majority of them, that is, 8/12 (67%). It has to be underlined that, when the interpretation was lacking information, a referral to genetic counseling was indicated in most reports, that is, 4/8 (50%).
3.3. Report Results
Table 5 shows information most commonly missing in the reports for all schemes.
3.3.1. Cystic Fibrosis
70% of reports assessed were correct and complete; 50 reports were not evaluated for lack of interpretation or for genotyping error. Few laboratories presented incomplete reports for lack of information such as clinical indication, and ethnic or geographic origin of the patient. Only one laboratory (4 samples) sent inadequate reports for lack or unclear reporting of important information.
3.3.2. Beta Thalassemia
65% of reports assessed were correct and complete; 1 report was not assessed for incorrect genotype. Few laboratories presented incomplete reports for lack of information such as clinical indication and title of the report; only one laboratory sent inadequate reports on all their samples for lack or unclear reporting of important information.
3.3.3. Fragile X Syndrome
Only 60% of reports were found to be correct and complete. Few laboratories presented incomplete reports for lack of information such as voice of clinical indication or indication of gender.
3.3.4. Familial Adenomatous Polyposis of the Colon (Gene APC)
60% of reports were found to be correct and complete. One laboratory (4 samples) did not report clinical indications; one laboratory (4 samples) did not indicate the title of the report.
4. Discussion
The EQA program focuses in particular on standardizing laboratory procedures. Participation in EQA schemes provides a measure of technical, analytical, and interpretative performance. It has an educational role for laboratories and gives the opportunity to review their internal standards and policies, and also to provide advice on the updating of best practice guidelines [1, 4, 21]. Moreover participation in EQA program is essential for laboratory accreditation with the international ISO 15189 standard [7, 21]. To date, EQA programs have been also used by laboratories as a tool for improving sample processing quality, hence the assessment takes into account not only genotyping and reporting, but it also looks at other aspects such as the interpretation of results [1, 21].
The Italian molecular EQA program was established in 2001; the VII round (2010) was reviewed versus previous rounds.
Fifty-six laboratories participated in the VII round and 320 samples were analyzed.
Genotyping was found to be quite satisfactory in general; only 4/56 laboratories in all schemes failed to correctly genotype samples: 3 for CF (4% of cases) and 1 for BT (1,5% of cases). In comparison, the error percentage for the CF scheme was higher in this round than the median error rate registered in previous rounds (0.2%) [14]. In the framework of the BT scheme, only the analysis of 1/68 samples was wrong (1.5%) and the error percentage was higher than the median error rate reported in previous rounds (0.33%) [16]. The analysis of these errors shows that techniques were generally well performed and raw data had good quality. In spite of previous rounds, where errors were mainly due to technical errors, during the VII round all errors were caused by suboptimal management of samples [14, 16].
No genotyping errors were performed by laboratories participating in the APC and FX schemes; this is a good result which reflects the outcome of the other rounds for the APC scheme and it represents an improvement in quality for the FX scheme [15, 17]. However, in the FX scheme, some genotypes were not completely defined because the number of triplets were not accurately reported in 28% of laboratory reports, and a variant, in one sample in APC scheme, was not detected by one laboratory.
Although laboratories returned acceptable analytical results in 97.5% of samples in all schemes, in the majority of reports, with variants within each schemes, the interpretation was correct but not complete. This data can be explained by the fact that during routine analysis, laboratories write out a technical report and leave it up to the genetic consultant to provide an accurate interpretation of genotype results in a separate report. In fact about 82% of laboratories who were penalized for lack of interpretation, or lack of necessary information, suggest and/or offer genetic counseling.
A comparison between our results and other published data shows a similar scenario during the first year of the interpretation survey [5, 22].
Like genotyping results, also report results were satisfactory for completeness of the information; during the VII round, in fact, 66% of reports on samples were good.
The focus on IEQA for the VII round shows satisfactory quality in genotyping and reports in comparison with previous published data, but we still observe room for improvement. A closer examination of the entire process of data reporting has highlighted the need to focus attention on the interpretation of results; this aspect was neglected in previous rounds because more attention was attached to the technical approach to sample analysis in order to ensure sound genotyping results.
All data collected within the framework of the IEQA and the European EQAs highlighted the need and the importance to carry on this activity in order to ensure adequate quality standards for the genetic tests performed in all laboratories.
At the present time the VIII round of the IEQA is under way; there has been an increase in the number of participating laboratories, and we expect some improvements in the laboratories that participated in the VII round.
Moreover in this round we are introducing a “poor performance” score for laboratories that make critical errors in genotype and/or interpretation and/or reports, that may significantly affect patient management. In this context our role is to contribute to improving laboratory quality through specific educational actions, improving the dialogue with laboratories and, if necessary, involving assessors and national experts in the process.
Acknowledgments
The authors thank the Galliera Genetic Bank—“Telethon Genetic Biobank Network,” supported by Italian Telethon grants (Project no. GTB07001A) for providing them with specimens. The authors are also grateful to the laboratories that participated in the IEQA for contributing their data to all schemes.