Don't Sell Your Guidelines Short - Remember to Report! (Part 2)

Part II: The Most Common Errors in Guideline Reporting (and Their Easy Fixes)

In Part I, we discussed the importance of reporting crucial details about the development process of a guideline and the various tools and checklists used to assess the quality of guidelines through what is reported.

In this post, we’ll provide a list of items that guidelines commonly neglect to report but which can easily and instantly improve the chances of a positive quality appraisal of a guideline. Remember: items that don’t have to do with the recommendations directly can always be described in detail elsewhere, such as in a supplemental document or a web site. Just make sure all materials are referenced in the guideline text so that its users – and those who may be assessing the guideline for research purposes – can easily find the relevant information.

1.     The target audience. Be explicit about who the guideline is and isn’t targeted toward. Rather than “healthcare professionals,” specifically list specialties and/or settings when appropriate (e,g., neurologists, primary care providers in low-income communities).

2.     The clinical questions. All good answers must begin with a well-formulated question, but it is surprisingly common for guidelines to leave out an explicit description of the key questions driving the guideline. Why was this guideline created? What questions are the recommendations addressing? Simply listing all questions in PICO (Population, Intervention, Comparison, Outcome) format at the beginning of the text and even as subheaders throughout the document will effectively guide readers through the text and provide helpful context for the appropriate application of recommendations.

3.     The patient voice.  The use of patient/consumer representation in the planning of a guideline and in drafting the resulting recommendations appears on many assessment tools and standard sets. If patients or consumers are involved in the development of a guideline at any part of the process, make sure to describe how and when – including whether educational strategies to increase their effective participation where used. Plans for dissemination of recommendations to the public beyond healthcare providers also apply here.

4.     The external review and public comment process. How were external reviewers selected? Were their identities kept confidential from the guideline authors? Did the guideline panel keep a formal documentation of all reviewers’ comments and respond to each one specifically? Lastly, was the guideline available for public comment, and how long was it available? The addition of these fine details is relatively easy but can drastically improve the assessment of a guideline in this domain.

5.     Implementation considerations. A perfect guideline is worth nothing if it isn’t followed. What are some potential barriers for appropriate implementation of the guideline of which its users should be aware? Have any tools been developed to improve the chances of successful implementation? How might resource availability affect implementation, and how should implementation be monitored?

6.     Conflicts of interest. This crucial aspect of guideline development goes further than simply listing interest disclosures of the authors. If any potential conflicts exist, describe how they were managed throughout the process. If the guideline has been funded by external sources, it’s important to note this and to describe the terms of the funding.

7.     Plans for update. Along with indicating the date of the relevant review of evidence, describe any plans and procedures for surveillance of the literature and updating of the guidelines. In addition, provide an expiration date for the current guideline.

Whether being read by a provider with the intention of implementing its recommendations or by a researcher assessing its quality, a guideline is only as good as how well it is written. Including as much information as possible, whether in the text of the guideline itself or in a referenced external source, will improve the chances of a fair appraisal of a guideline’s quality – and, more importantly, its successful implementation.

Spring 2020 - Scholarship Recipients

Earlier this month, the U.S. GRADE Network welcomed 51 participants to the Twelfth GRADE Guideline Workshop in sunny Phoenix, Arizona. Two participants, Giulia Lane and Hayley Dunnack, were awarded a scholarship from the Evidence Foundation that covered their workshop registration fee. As the two latest additions in a line of over 40 Foundation scholars since 2014, Lane and Dunnack presented to their fellow participants about their projects related to improving the use of evidence-based decision-making in healthcare.

Hayley Dunnack, left, and Giulia Lane, right, were the recipients of the Evidence Foundation scholarship for the spring 2020 GRADE workshop held in Phoenix, AZ

Hayley Dunnack: Opportunities and Challenges in Guidelines for Nursing Practice

Hayley Dunnack, BSN, CMS-RN, OCN is a PhD candidate at the University of Connecticut School of Nursing. Dunnack presented on her work with the Oncology Nursing Society (ONS) to develop guidelines on symptom management for patients with cancer. Currently, ONS has several projects underway to develop guidelines on symptom management based on systematic reviews of the evidence and using GRADE guideline methodology. A current challenge in the ONS’ guideline development relates to the development of guidelines on topics for which randomized studies are sparse or difficult to conduct, such as the management of hypersensitivity reactions or the use of vascular access devices.  Dunnack’s project aims to examine the use of non-randomized and “real-world” evidence to inform the development of rigorous guidelines based on systematic reviews and using the GRADE process.

“I had an incredible experience at the GRADE Guideline Development Workshop,” said Dunnack. “From the didactic material to the small group sessions, there was so much to learn. The ability to network with other professionals and hear their viewpoints added immensely to the experience."

Dunnack and Lane pose with Dr. Shahnaz Sultan of the U.S. GRADE Network

Giulia Lane: Shared Decision-Making in Urology Guidelines

Dr. Giulia Lane is a urology fellow at the University of Michigan. Lane’s project examines the current uptake and opportunities for improvement in the use of shared decision-making aids in the field of urology. Although shared decision-making aids are a critical step of assessing patients’ values and preferences, Lane explained, they are not commonly used in the field of urology. While almost everyone completes the initial step of informing patients about treatment options, exploring individual preferences is less frequent, and of the 29 American Urological Association (AUA) guidelines reviewed by Dr. Lane, only 12 mention shared decision-making at all. Of those guidelines that mentioned shared decision-making by name, only seven (58%) further described the process. Clinicians in urology, Dr. Lane concluded, are not provided support or encouragement to use shared decision-making in their practice. However, this appears to be changing: shared decision-making was mentioned in 2019 guidelines nearly eight times as often as those published in 2010.

In contrast to what is commonly believed, shared decision-making does not take more time once clinicians are trained. Clinicians should consider providing ready-made online modules to patients to review ahead of their visit, further reducing the time required to improve patient engagement and understanding of their treatment options. Dr. Lane’s work provides a baseline assessment of the use of shared decision-making in urology and will help inform the direction of future research in this area.

“I felt that the best part of the GRADE workshop was the small-group setting with intimate access to experts in the field,” said Dr. Lane. “The networking opportunities were exceptional and I met several people with whom I plan to collaborate. The people I met at the GRADE workshop and the transfer of ideas were the most important take away point for me.”

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Chicago, Illinois will be July 31, 2020.

Rapid Guidelines in GRADE Pt. III: A checklist for rigorously rapid recommendations

In recent posts, we have introduced the concept of rapid recommendations as well as how developers of these recommendations at the World Health Organization (WHO) perceive facilitators and barriers to this process. This information was gathered as part of a published series on rapid guidance in 2018.

In the final part of the series, Morgan and colleagues propose an extension of the G-I-N/McMaster Checklist for Guideline Development aimed at those producing rapid guidelines. Comprising 21 discrete guideline principles that align with the original Guideline Development Checklist, the checklist is a tool for developers to take stock of the resources available for their rapid guideline development goals and identify areas in need of improvement. Important considerations include:

·      Make use of virtual meetings (Principle 7) and pre-meeting voting (Principle 15) to expedite the drafting of recommendations.

·      If possible, limit guideline panel composition to those not reporting financial COIs, or if not possible to provide necessary topic expertise, transparently declare any modifications of existing COI policies for the topic at hand (Principle 9).

·      Limiting the number of PICOs (Principle 10) and limiting the assessment of outcomes to only those deemed critical (Principle 11).

·      Consider ways to facilitate systematic review stage, such as updating existing reviews, developing rapid reviews, or tailoring search criteria to a smaller scope (Principle 13)

·      Arrange for external reviewers early on in the process so that they are quickly deployed when a draft is available for review (Principle 18).

The full checklist extension for rapid guidelines can be viewed here.

Morgan, R.L., Florez, I., Falavigna, M. et al. Development of rapid guidelines: 3. GIN-McMaster Guideline Development Checklist extension for rapid recommendations. Health Res Policy Sys 16, 63 (2018). https://doi.org/10.1186/s12961-018-0330-0

Manuscript available at the publisher's website here.

Rapid Guidelines in GRADE Pt. II: Rapid Recs in the Real World

In Part I of our series on rapid guidelines, we discussed the utility and terminology of rapid recommendations: those recommendations made in response to an urgent public health issue with timeframes ranging from a few short hours up to three months' time.

Who develops rapid guidelines?

In the first of a three-part series on rapid guidance published in 2018, Kowalski and colleagues conducted a systematic survey of the methodologies and processes of rapid guideline-producing organizations. Nomenclature used to identify these documents varied by organization, from “rapid advice guideline” to “interim guidance” to “short clinical guideline.” While the quality of these documents as assessed with the AGREE II tool was variable, it was greater in documents from the WHO and NICE than it was for the CDC or other smaller organizations. While NICE guidelines were of higher quality as assessed with the domains of AGREE II, they took substantially more time to develop than those from WHO.

It's important to note that while terminology differs between organizations, the word "interim" has been used to connote a response to an emergent public health issue with shorter time frames than a rapid guideline - typically on the order of 1-3 weeks. 

Organization	Nomenclature	AGREE II domain score range (lowest – highest)	Development timeline (from manual)
WHO	Rapid advice guideline	54 - 92	1-3 months
NICE	Short clinical guideline	81 - 94	11-13 months
CDC	Interim guidance	10 - 82	Not reported
Other	Interim guidelines, interim position statement, clinical guidelines	21 - 67	Not reported

Common Challenges and Facilitators to Rapid Guideline Development

While both the World Health Organization (WHO) and the National Institute for Health and Care Excellence (NICE) reported the use of a systematic review of the evidence to guide recommendations, common issues among developers included a lack of reporting on the management of conflict of interest, external review, and of the process for the drafting of recommendation.

In follow-up qualitative interviews with guideline-developing staff from WHO, participants cited a lack of adequate staffing, monetary resources, and evidence as key obstacles to the development of rapid guidelines. While the development of a systematic review is likely one of the more time-consuming elements of a rapid guideline process, most participants agreed that it is a fundamental part of developing trustworthy guidance that should not be skipped if possible. 

Participants also indicated that the external/peer review process can add unwanted time to the development of rapid guidelines. To this effect, developers can consider limiting the reach of peer review to the final draft only as well as reducing the ability to drastically change recommendations in a way that would require a reconvention of the guideline panel. Virtual conferencing technology was named as a facilitator to developing guidelines on a quicker schedule by reducing the need for face-to-face meetings. 

For a checklist to guide the development of rapid recommendations, see the G-I-N/McMaster checklist extension for rapid guidelines.


Kowalski, S.C., Morgan, R.L., Falavigna, M. et al. Development of rapid guidelines: 1. Systematic survey of current practices and methods. Health Res Policy Sys 16, 61 (2018).

Manuscript available at the publisher's website here.  

Florez, I.D., Morgan, R.L., Falavigna, M. et al. Development of rapid guidelines: 2. A qualitative study with WHO guideline developers. Health Res Policy Sys 16, 62 (2018).

Manuscript available at the publisher's website here.  

Rapid Guidelines in GRADE Pt. I: Needed Advice when Time is of the Essence

While most clinical practice guidelines take 2-3 years to develop and publish, the emergence of a public health crisis or urgent humanitarian need requires the dissemination of evidence-based guidance in a more rapid manner. To this effect, several national- and international-level guideline-producing organizations, such as the Centers for Disease Control (CDC) and the World Health Organization (WHO), have developed processes for the development of evidence-based guidance for these more urgent situations.

WHO’s 2006 recommendations for the pharmacological management of avian influenza in humans is one example of a rapidly developed guideline. Of current relevance, WHO has recently published interim guidance on the management of severe acute respiratory infection when novel coronavirus is suspected, and the UK’s National Institute for Health and Care Excellence (NICE) has also developed interim guidelines for the treatment of COVID-19 in patients receiving critical care, kidney dialysis, and systemic anticancer therapy. Because this matter is rapidly evolving and advice is needed immediately, the protocols used by NICE, WHO and other organizations are different than it would be for less urgent topics.

Can rapid guidelines use GRADE?

In short, yes. Recommendations can be made based on the transparent grading and reporting of the certainty of evidence that lie at the heart of GRADE, whether this is over the timeframe of hours, days, weeks, or months. The key word here is transparent: no matter the speed of development, recommendations should always be couched within the terms of the certainty of evidence behind them, and judgments of the evidence should be clearly presented. In a 2016 paper on the use of GRADE to respond to health questions with different levels of urgency, Thayer and Schünemann provide terms for the various speeds of response, and considerations for recommendations therein:

• Ultra-short emergency response: 1 or more hours

• Urgent response: 1-3 weeks

• Rapid response: 1-3 months

• Routine response: More than three months

Recommendations can still be formed based on the certainty of the evidence that's available, whatever that evidence may be. While systematic reviews of all available evidence are a foundational aspect of non-urgent guidelines, evidence in the form of narrative syntheses, modeling, or late-breaking data from the field can be used when time is short and systematically compiled data are sparse. Regardless of the source, the domains of GRADE still allow for evidence to be appraised and to guide the resulting direction and strength of recommendations.

Stay tuned for Pt. II coming soon, where we'll take a closer look at organizations that have developed rapid recommendations in response to time-sensitive public health issues.

For a checklist to guide the development of rapid recommendations, see the G-I-N/McMaster checklist.

For more information about appraising the certainty of evidence in the lack of meta-analyzed data, see this paper.

Thayer KA & Schünemann H. Using GRADE to Respond to Health Question With Different Levels of Urgency. Environment international. 2016 July-August: 585-589.

Manuscript available at the publisher's website here.