Abstract

Purpose. To evaluate the methodological quality of age-related macular degeneration (AMD) clinical practice guidelines (CPGs). Methods. AMD CPGs published by the American Academy of Ophthalmology (AAO) and Royal College of Ophthalmologists (RCO) were appraised by independent reviewers using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument, which comprises six domains (Scope and Purpose, Stakeholder Involvement, Rigor of Development, Clarity of Presentation, Applicability, and Editorial Independence), and an Overall Assessment score summarizing methodological quality across all domains. Results. Average domain scores ranged from 35% to 83% for the AAO CPG and from 17% to 83% for the RCO CPG. Intraclass correlation coefficients for the reliability of mean scores for the AAO and RCO CPGs were 0.74 and 0.88, respectively. The strongest domains were Scope and Purpose and Clarity of Presentation. The weakest were Stakeholder Involvement (AAO) and Editorial Independence (RCO). Conclusions. Future AMD CPGs can be improved by involving all relevant stakeholders in guideline development, ensuring transparency of guideline development and review methodology, improving guideline applicability with respect to economic considerations, and addressing potential conflict of interests within the development group.

1. Introduction

Blindness caused by age-related macular degeneration (AMD) affects an estimated 32.4 million people worldwide, with an additional 191 million people suffering from moderate and severe AMD-related vision impairment [1]. Therefore, it is critical to ensure that clinical practice guidelines (CPGs) for AMD abide by rigorous standards of development. However, recent studies have reported significant shortcomings in the quality of CPGs, such as a failure to clearly delineate guideline methodology, poor evidence quality, and lack of conflict of interests disclosures among guideline development group members [2–6]. In addition, there has been no recent evaluation of CPGs for AMD.

The Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument is a valid and reliable tool for assessing the methodological quality of CPGs [7–9]. The original AGREE rubric along with the updated AGREE II instrument have previously been used to assess methodological quality of CPGs for the management of cataract and primary open-angle glaucoma, with shortcomings noted in a number of areas including Stakeholder Involvement, transparency of methodology, and Editorial Independence [10, 11]. We used the AGREE II instrument to evaluate AMD CPGs developed by the American Academy of Ophthalmology (AAO) and Royal College of Ophthalmologists (RCO).

2. Materials and Methods

The AGREE II instrument consists of 23 assessment items, with evaluators score on a scale of one (strongly disagree) to seven (strongly agree). The 23 items are organized into six quality domains: (1) Scope and Purpose; (2) Stakeholder Involvement; (3) Rigor of Development; (4) Clarity of Presentation; (5) Applicability; and (6) Editorial Independence [8]. Two final summary assessment items in the AGREE II rubric—the Overall Assessment score and reviewer recommendation of the guideline—enable reviewers to holistically evaluate each CPG.

The AAO CPG outlines recommendations for the diagnosis, treatment (nonneovascular and neovascular AMD), and follow-up care for adult AMD patients [12]. The RCO CPG outlines recommendations for AMD diagnosis, therapies for acute neovascular AMD, treatment delivery, management of nonneovascular AMD, management of chronic/long standing vision loss, referral pathways, and AMD research [13]. Four of the authors (AMW, CMW, BKY, and DJW) independently evaluated each CPG. The scores were then averaged and summarized as scaled percentage scores using the following formula given by the AGREE II: (Obtained Score − Minimum Possible Score)/(Maximum Possible Score − Minimum Possible Score) [8]. An intraclass correlation coefficient (ICC) was used to measure interrater agreement for each guideline’s scores.

3. Results

Domain scores are summarized in Table 1; mean scores per evaluator were averaged across all items in each domain. The AAO CPG scored lowest in Domain 2 (Stakeholder Involvement) with an average score of 35% and highest in Domain 1 (Scope and Purpose) and Domain 4 (Clarity of Presentation) with scores of 83% and 79%, respectively; the average Overall Assessment was 4.5 out of 7. The RCO CPG scored lowest in Domain 6 (Editorial Independence) with 17% and highest in Domain 2 and Domain 4 with scores of 83% and 74%, respectively; the average Overall Assessment was 4 out of 7.

ICCs for the AAO and RCO CPG scores were 0.74 and 0.88, respectively. Strengths and weaknesses of the guidelines are summarized in Table 2, derived from consensus among at least three of the four reviewers in consideration of the average item scores and evaluator comments.

4. Discussion

To our knowledge, there have been no previous studies critically assessing multiple AMD CPGs using the AGREE II instrument. Our analysis with the AGREE II instrument outlined a number of methodological shortcomings in the guidelines.

With respect to Stakeholder Involvement (Domain 2), both CPGs failed to describe participation of patient representatives, if any, in the guideline development process. While the RCO guideline cited the involvement of one patient representative in the guideline development group, it was not possible to determine if patient groups were consulted in the development of the AAO guideline. Neither CPG detailed outcomes of public review nor how reviewer comments were used to guide the process of guideline development. An additional criterion within Domain 2 is the requirement for full disclosure of panel members’ individual roles and areas of expertise. Although panel members’ occupational degrees and areas of expertise were provided, CPG did not delineate panel members’ specific roles within the guideline development group. Similar weaknesses in stakeholder involvement were described in the studies assessing CPGs for the management of cataract and primary open-angle glaucoma [10, 11]. Ensuring that the guideline development panel includes representatives of all stakeholders is critical in order to allow for full inclusion of relevant scientific evidence and prevent recommendation biases in favor of a particular specialty or treatment option [14]. Additionally, it has been shown that inclusion of patient views in guideline development has crucial implications for the success of guideline implementation [15, 16].

The AMD CPGs had a few shortcomings in Rigor of Development (Domain 3), the largest AGREE II domain. First, both CPGs failed to disclose details concerning external review methods, including feedback gathered from external review, patient input if any, and how the outcomes informed guideline development. Second, the CPGs lacked detailed search terms, explicit criteria for selecting evidence, names of databases searched (AAO), and dates of search (RCO). Additionally, while the AAO CPG provided an explicit rating system indicating the quality of each primary source, the RCO CPG lacked such information. Finally, the RCO CPG lacked primary source citations following each recommendation, rendering the link between evidence and recommendations unclear. Consistent citation of evidence and disclosure of evidence search methods should be incorporated into AMD CPGs to ensure rigor and transparency of evidence-based recommendation development.

Regarding Clarity of Presentation (Domain 4), recommendations were often difficult to identify within the body of each guideline due to the fact that they were either embedded within paragraphs (AAO) or listed without any direct citations of evidence or grade level indicating recommendation strength (RCO). Improving Clarity of Presentation is important as this facilitates guideline interpretation and implementation.

In Applicability (Domain 5), a common shortcoming was the lack of research on cost information and description of how economic considerations guided recommendation development. While the RCO CPG mentioned cost considerations as important factors affecting treatment adherence, few provisions were given regarding specific ways to address economic barriers to the implementation of particular recommendations in either guideline. Inclusion of cost and resource information within CPGs can facilitate transparency of treatment selection and is critically important in light of medical technology advancement amid rising healthcare costs [17]. In addition, although the AAO CPG provided summary tables, patient education materials, and a list of major recommendations to aid their implementation, the RCO CPG failed to supply such resources. The provision of such tools is critical to facilitate guideline accessibility.

Finally, the AMD CPGs had several gaps in Editorial Independence (Domain 6). Neither CPG delineated methods by which commercial interests were sought nor addressed how potential conflicts of interest (COIs) may have influenced guideline development. The AAO provides a statement on its website regarding its policy on industry relationships within clinical guideline committees [18]; however, this document does not specifically address how commercial relationships within the AAO AMD CPG development group could have influenced guideline development. Furthermore, while the AAO CPG disclosed COIs for all panel members, the RCO CPG only listed COIs of the chair. Finally, the RCO guideline did not clearly state the independence of the funding body nor describe its potential influence on guideline development. Lack of COI transparency is of particular concern when development group chairs and/or a majority of members have financial interests that are not addressed, as was the case for both AMD development groups. As potential conflicts may result in biased reporting, it is critical to ensure editorial independence and transparent COI disclosure of guideline development group members [19].

There were several potential limitations to this study. First, any rating instrument may lack objectivity. However, the AGREE II instrument has been proven to be reliable and valid, particularly with four independent reviewers. In this study, the independent reviewers were highly reliable (ICCs of 0.74 and 0.88), which is consistent with the existing research that supports the validity of AGREE II as an assessment tool [20–22]. Second, appraisals within domains such as Stakeholder Involvement, Applicability, and Editorial Independence are unable to distinguish between guidelines that gathered information and did not disclose the data from guidelines that failed to obtain any data pertaining to those domains. Although each group may have internally investigated conflict of interests and/or economic factors affecting guideline recommendations, a failure to disclose such information impairs the transparency of guideline methodology. Finally, the AGREE II assigns equal weight to each of the six domains. Since domains vary in the number of items and evaluation criteria, we used the Overall Assessment score and evaluator comments to inform comprehensive appraisals of each guideline.

5. Conclusions

In summary, the AGREE II evaluation of AMD CPGs highlighted methodological weaknesses in the domains of Stakeholder Involvement, Rigor of Development, Applicability, and Editorial Independence. We recommend a number of improvements for future guidelines, using whenever possible one of the two CPGs reviewed herein as a model (Table 3). Regular assessment of CPGs is important to ensure methodological rigor and quality development of guidelines offering evidence-based recommendations for clinical practice.

Presentation

Poster presentation at ARVO 2015.

Disclaimer

The views expressed in this paper are those of the authors’ and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work was supported in part with resources and the use of facilities at the Providence VA Medical Center, Providence, Rhode Island, USA.