Don't Sell Your Guidelines Short - Remember to Report! (Part 2)

Part II: The Most Common Errors in Guideline Reporting (and Their Easy Fixes)

In Part I, we discussed the importance of reporting crucial details about the development process of a guideline and the various tools and checklists used to assess the quality of guidelines through what is reported.

In this post, we’ll provide a list of items that guidelines commonly neglect to report but which can easily and instantly improve the chances of a positive quality appraisal of a guideline. Remember: items that don’t have to do with the recommendations directly can always be described in detail elsewhere, such as in a supplemental document or a web site. Just make sure all materials are referenced in the guideline text so that its users – and those who may be assessing the guideline for research purposes – can easily find the relevant information.

1.     The target audience. Be explicit about who the guideline is and isn’t targeted toward. Rather than “healthcare professionals,” specifically list specialties and/or settings when appropriate (e,g., neurologists, primary care providers in low-income communities).

2.     The clinical questions. All good answers must begin with a well-formulated question, but it is surprisingly common for guidelines to leave out an explicit description of the key questions driving the guideline. Why was this guideline created? What questions are the recommendations addressing? Simply listing all questions in PICO (Population, Intervention, Comparison, Outcome) format at the beginning of the text and even as subheaders throughout the document will effectively guide readers through the text and provide helpful context for the appropriate application of recommendations.

3.     The patient voice.  The use of patient/consumer representation in the planning of a guideline and in drafting the resulting recommendations appears on many assessment tools and standard sets. If patients or consumers are involved in the development of a guideline at any part of the process, make sure to describe how and when – including whether educational strategies to increase their effective participation where used. Plans for dissemination of recommendations to the public beyond healthcare providers also apply here.

4.     The external review and public comment process. How were external reviewers selected? Were their identities kept confidential from the guideline authors? Did the guideline panel keep a formal documentation of all reviewers’ comments and respond to each one specifically? Lastly, was the guideline available for public comment, and how long was it available? The addition of these fine details is relatively easy but can drastically improve the assessment of a guideline in this domain.

5.     Implementation considerations. A perfect guideline is worth nothing if it isn’t followed. What are some potential barriers for appropriate implementation of the guideline of which its users should be aware? Have any tools been developed to improve the chances of successful implementation? How might resource availability affect implementation, and how should implementation be monitored?

6.     Conflicts of interest. This crucial aspect of guideline development goes further than simply listing interest disclosures of the authors. If any potential conflicts exist, describe how they were managed throughout the process. If the guideline has been funded by external sources, it’s important to note this and to describe the terms of the funding.

7.     Plans for update. Along with indicating the date of the relevant review of evidence, describe any plans and procedures for surveillance of the literature and updating of the guidelines. In addition, provide an expiration date for the current guideline.

Whether being read by a provider with the intention of implementing its recommendations or by a researcher assessing its quality, a guideline is only as good as how well it is written. Including as much information as possible, whether in the text of the guideline itself or in a referenced external source, will improve the chances of a fair appraisal of a guideline’s quality – and, more importantly, its successful implementation.

Extremely Serious Research Short: GRADE’s terminology for rating down by three levels

Contributed by Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

Since the inception of GRADE two decades ago, GRADE methodology has needed to evolve along with the arrival of new ways of assessing the evidence. One such evolution has come with the introduction of methods for assessing risk of bias for non-randomized studies, such as the Risk Of Bias In Non-randomized Studies (ROBINS-I) and the RoB Instrument for Nonrandomized Studies of Exposures (ROBINS-E).

Because these tools assess the risk of bias in non-randomized studies as if they represent a pragmatic trial, they automatically begin from a lower risk of bias than alternative assessments such as the Newcastle-Ottowa Scale. When rating down in GRADE, however, non-randomized studies start as low certainty of evidence before any rating up or down occurs. This means that while a study assessed with ROBINS-I or E would start as high-quality evidence, it may require a reduction of three levels if very serious risk of bias is present. In other words, a reduction of three levels for a study assessed with ROBINS-I or E would be analogous to a two-level reduction for a non-randomized study assessed with another method.

A rating by any other name…

In order to determine what exactly this new three-level reduction should be called, members of the GRADE Working Group conducted a survey of 225 participants recruited via social media, the Guidelines International Network (G-I-N), and other sources. Just over one-third (34.2%) were members of the GRADE Working Group and all respondents had participated in guideline development in some capacity. The results are presented in a newly published article as part of a new “GRADE Notes” series in the Journal of Clinical Epidemiology.

Within the survey, participants were asked to rate the following terms for this novel three-level reduction, from least (1) to most-favored (4):

• Critically serious
• Extremely serious
• Most serious
• Very, very serious

Respondents' average ranking of terms. 

T. Piggott et al. / Journal of Clinical Epidemiology - (2020)

“Extremely serious” took the lead as the most favorably ranked term with an average score of 3.19, with “critically serious” a close second at 3.12. Respondents found “extremely serious” the most agreeable due to its clarity and the fact that it seemed to “naturally” follow the existing two-level term, “very serious.”

The term “extremely serious” can now be found within the GRADEpro application when rating the certainty of evidence within non-randomized studies while utilizing the ROBINS-I or ROBINS-E instruments.

Piggott T, Morgan RL, Cuello-Garcia CA, Santesso N, Mustafa RA, Meerpohl JJ, Schünemann HJ, GRADE Working Group. GRADE notes: Extremely Serious, GRADE’s Terminology for Rating Down by 3-Levels. Journal of Clinical Epidemiology. 2019 Dec 19.

Manuscript available here on publisher's site.

Research Shorts: U.S. Guideline Developers Inconsistently Applying Criteria for Appropriate Evidence Grading

Contributed by Philipp Dahm, MD, MHSc, FACS

Guideline Developers in the United States were Inconsistent in Applying Criteria for Appropriate GRADE Use


Our study was motivated by the anecdotal observation that many US-based organizations appeared to be endorsing the GRADE approach but did not necessarily apply it to the fullest extent. We therefore sought to formally study this issue applying six published criteria of appropriate GRADE use. We limited to search to guidelines from US-based organizations that were included in the National Guideline Clearinghouse (NGC) which implied that they met certain, minimal criteria for evidence-based guidelines. Our search reached back to January 2011 and went to June 2018 after which time the NGCH lost its funding and stopped existing in that form.

Among guidelines documents from 315 organizations included in the database, 135 were from the US and were represented by at least one guideline. Our analysis ultimately included 67 guideline documents from 44 organizations. The vast majority of these guidelines were from professional organizations; mostly related to the field of internal medicine and its subspecialties. With regard to domains for rating the certainty of evidence, only one in 10 was explicit about including all five criteria for downgrading (study limitations, indirectness, inconsistency, imprecision, and publication bias) for a body of evidence from randomized trials and all three domains (large magnitude of effect, dose-response gradient, and direction of residual bias) for rating up a body of evidence from non-randomized trials. Over half of guidelines described explicit consideration of all four central domains (certainty of evidence, balance of benefits to harms, patients’ values and preferences and resource utilization) for moving from evidence to recommendations. All guidelines included the certainty of evidence and the vast majority also addressed the balance of desirable and undesirable consequences. When comparing guidelines published in 2011-2014 versus 2015-18, rates of appropriate use were higher for nearly all criteria, but only one main criterion met statistical significance, namely the reporting of evidence summaries supporting recommendations.

The take-home messages from this study are that one-in-three US based organizations developing evidence-based guidelines report the use of GRADE but that adherence to published criteria is quite inconsistent. As GRADE finds increasing uptake worldwide, continued efforts in training guideline methodologists and panel members will be important to assure appropriate application of GRADE methodology.

Dixon C, Dixon PE, Sultan S, Mustafa R, Morgan RL, Murad MH, Falck-Ytter Y, Dahm P. Guideline Developers in the United States were Inconsistent in Applying Criteria for Appropriate GRADE Use. Journal of Clinical Epidemiology. 2020 Mar 4.

Don’t Sell Your Guideline Short – Remember to Report! (Part 1)

Contributed by Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

The development of a high-quality, evidence-based clinical guideline is no small feat. It requires significant time and effort from content experts, methodologists, and organizational staff and typically takes more than 1-2 years from start to finish.

Given the effort and hours that go into guideline development, it’s all too easy - and all too common - for the reporting of the development process of these guidelines to significantly undersell their quality. This is important, because published analyses assessing the quality of guidelines will likely only use what is reported or referenced in the text of the guideline. In other words, guidelines that do not adequately report on the methods they used to develop their recommendations will be under-appraised in the published literature – and this could lead to a gross underestimation of a guideline-developing organization’s work as a whole.

Quality and Reporting Standards: A Brief Review

Over the past decade, a number of standard sets, reporting checklists, and appraisal tools have been published to assist guideline developers in the reporting of their methods and to provide ways for researchers to assess the quality of these guidelines. These standards and methods of appraisal include but are not limited to:

• The Appraisal of Guidelines for Research and Evaluation (AGREE) II tool (2010)
• the National Academy of Medicine (formerly the Institute of Medicine [IOM]) Standards for Trustworthy Clinical Practice Guidelines (2011)
• the Guideline International Network (G-I-N) Key Components of High-Quality and Trustworthy Guidelines (2012)
• World Health Organization (WHO) Handbook for Guideline Development (2nd ed., 2014)
• Reporting Items for practice Guidelines in HealThcare (RIGHT) Statement (2017)

Report, or it didn’t happen.

A guideline may be developed using the most water-tight, rigorous methods, but if these methods are not adequately described either in the text of the guideline or in a referenced external text, then an assessor will likely under-appraise the quality of a guideline. To ensure the most accurate appraisal of a guideline possible, guideline developers should consider the following helpful tips:

• Create a guideline template including boilerplate text that meets as much reporting criteria as possible, such as a general description of the systematic review and recommendations development processes; competing interest statements for all involved authors and guideline panel members; a description of the method used to assess certainty of evidence and grade the strength of recommendations; and a clear table at the beginning of the document listing all clinical questions and resulting recommendations.
• Maintain an up-to-date, in-depth description of the guideline development process on the website of the guideline-producing organization. Refer to this page specifically in the text of the guideline. This allows both guideline end-users and potential assessors to view the development process in depth without requiring too much space in the guideline document itself. 
• When in doubt, refer it out. If there are supplemental texts to the guideline that include information related to the development process – such as an underlying systematic review or a list of authors’ conflict of interest disclosures – make sure these documents are clearly referenced in the guideline text and made easily accessible in the online version via hyperlinks. 
• Don’t make assumptions. Even aspects of the development process that seem obvious, such as whether the guideline is externally reviewed, will likely not be included in a published quality assessment if it is not explicitly mentioned. 
• Always be specific. Do not make the end-user of a guideline have to guess who the guideline is for, the clinical questions driving the guideline, or the appropriate scenarios in which to employ the recommendations. Utilizing the PICO (Population, Intervention, Comparison, Outcome) format to explicitly describe the clinical questions and resulting recommendations is a failsafe way to ensure your guideline is specific enough to be useful. 

Stay tuned for Part II where we provide a list of commonly overlooked items in published guidelines and discuss how to instantly improve the quality assessment of a guideline.