GRADE Takes on the Planet: Introducing GRADE Guidance 46, Integrating Planetary Health in Health Guidelines

For decades, health guideline developers using the GRADE framework have worked to ensure that recommendations reflect the best available evidence on patient outcomes, values, costs, and equity. A major new addition to that framework asks guideline panels to consider something broader: the health of the planet itself. In a paper recently published in Annals of Internal Medicine, Piggott, Saadat, Herrmann, and colleagues from the GRADE Planetary Health Project Group present the first official GRADE guidance for integrating planetary health into health guideline development.

Health systems are not passive bystanders in the planetary health crisis; they actively contribute to it. Health care accounts for an estimated 5.2% of global greenhouse gas (GHG) emissions, and its impacts on freshwater use may be even more significant relative to national benchmarks. Until now, however, health guidelines have rarely systematically considered these environmental consequences. A scoping review cited in the paper found just four health guidelines that had addressed planetary health dimensions at all, and even those did so with limited methodological rigor. This guidance is designed to close that gap.

The project group, working from January 2023 through May 2025, developed 7 domains of guidance through iterative case study analyses, expert workshops, and a two-round global Delphi consensus process involving 85 international experts. The result is a structured framework that maps onto the familiar steps of guideline development, from scoping and question formulation through to evidence synthesis and the evidence-to-decision (EtD) framework. 

The most concrete addition is a new criterion within the GRADE evidence-to-decision (EtD) framework, which is the table guideline panels use at the final stage of a review to move from evidence to recommendations. Alongside existing criteria like "balance of effects," "costs," and "equity," panels now have a dedicated row for planetary boundaries. For each boundary relevant to their guideline topic, panels work through a set of signaling questions drawn directly from environmental science, then record a summary judgment ranging from "large desirable" to "large undesirable." 

Four case studies bring the guidance to life, ranging from the choice between anesthetic gases (sevoflurane versus desflurane, with dramatically different GHG profiles) to dietary recommendations around red meat consumption. In the red meat case, applying the planetary health lens would have reversed the direction of a previously published conditional recommendation, shifting it from continuing current intake to reducing it, based on the intervention's large undesirable effects on climate, land use, freshwater, and nutrient cycles. This underscores that planetary health evidence can and should be considered when making recommendations.

This guidance is meant to fit into the existing GRADE framework, ensuring a seamless adoption. The framework provides signalling questions for each planetary boundary to help panels make informed judgments without requiring in-house environmental science experts. Including a planetary health expert on the panel is suggested but optional and the guidance is clear that not every guideline needs to address every planetary boundary. The goal is transparency and consideration.

As the authors note, the crises facing our planet warrant the best effort guideline panels can offer, and this framework gives them the tools to start.

References

Thomas Piggott, Pakeezah Saadat, Alina Herrmann, et al. Integrating Planetary Health in Health Guidelines (GRADE Guidance 46). Ann Intern Med 2026;179:874-884. [Epub 12 May 2026]. doi:10.7326/ANNALS-25-04761

https://www.acpjournals.org/doi/10.7326/ANNALS-25-04761 

Extra Resources

GRADE guidance articles in the Journal of Clinical Epidemiology (JCE) series: https://www.cochrane.org/learn/courses-and-resources/cochrane-methodology/grade/jce-series

GRADE Book is the most comprehensive and up-to-date resource on the GRADE approach. It will replace the previous GRADE Handbook by 2026, with updated content being progressively released. https://book.gradepro.org/

GRADE working group: https://www.gradeworkinggroup.org/

A New Template for Standardized Wording when Reporting Evidence-to-Decision Considerations in Guidelines

One of the major tenets of GRADE is that certainty of the evidence is just one component of decision-making. Ultimately, decision-makers also need to take into account important factors such as values and preferences, feasibility, and considerations of the impact of a decision on health equity and resource utilization. These factors and others are part of the Evidence-to-Decision (EtD) framework that guides the process from the assessment of certainty of evidence to the final formulation of recommendations in a structured, transparent manner.

Often, multiple teams and individuals involved in the development of a guideline will need to work together to compete the EtD process, which can be a source of confusion. Additionally, until now, no official guidance existed for the use of standardized wording when considering and reporting each EtD framework component. Earlier this year, Piggott and colleagues aimed to address this issue with an article published in the Journal of Clinical Epidemiology.

The project, comprising ten guideline development groups and over 250 recommendations, set out to develop a standardized framework for clear, transparent, and efficient wording when reporting Evidence-to-Decision components within a guideline. This template was then used in two guidelines in development - the European Commission Initiative on Breast Cancer (ECIBC) and the Endocrine Society guidelines on hyperglycemia, hypoglycemia and hypercalcemia. During this process, the authors were able to pilot the wording, receive feedback, and refine the template. The real-life guidelines were also used to provide examples of wording recommendations.

The article includes suggested wording structure and examples for reporting the magnitude and certainty of effect estimates, for conclusions of each portion of the EtD framework, and for justification of recommendations as well as notes on implementation considerations, monitoring and evaluation, and research priorities. 

The authors note that these suggestions are preliminary and may require further refinement. Additionally, current examples of consistent and clear wording of EtDs continues to be lacking, though the dissemination of this guidance may improve future publications. While the suggestions within the article are focused on clinical decisions related to management of conditions, future efforts may expand this to guidelines for diagnostic testing, coverage, and other important areas.

Piggott, T., Baldeh, T., Dietl, B., Wiercoch, W., Nieuwlaat, R., Santesso, N., ... & Schünemann, H. (2022). Standardized wording to improve efficiency and clarity of GRADE EtD frameworks in health guidelines. J Clin Epidemiol (online ahead of print). Manuscript available at the publisher's website here.

   

Guideline Development Resource Alert: the G-I-N Public and Patient Toolkit

One of the most common challenges to developing rigorous and high-quality guidelines is the inclusion of the patient and public perspective into the formulation of recommendations. In fact, in a recently published needs assessment of guideline developers worldwide, 81.5% answered with a 5 or greater on a 7-point Likert scale that the incorporation of the patient voice was a relevant need for their organization. 

Now, the Guidelines International Network (G-I-N) has launched a large-scale toolkit aimed to address commonly experienced issues related to patient and public involvement. The result of a combination of international experiences and best practice examples, the toolkit is a one-stop shop spanning the systematic review and guideline development process, from conducting targeted consultation with the public to recruiting and supporting patient panel members to communicating recommendations to the public at-large. As a "living resource," the toolkit will continue to expand and evolve as further information and experience is cultivated.

You can find the freely available toolkit here.

UpPriority: A new tool to guide the prioritization of guideline update efforts

The establishment of a process for assessing the need to update a clinical guideline based on new information and evidence is a key aspect of guideline quality. However, given limited time and resources, it is likely necessary to prioritize clinical questions that are most in need of an update from year to year. A new paper demonstrates proof of concept for the UpPriority Tool, which aims to allow guideline developers to prioritize questions for guideline update. 

The tool comprises six different items when assessing the need to update a given recommendation or topic of guideline:

• the potential impact of an outdated guideline on patient safety;
• the availability of new, relevant evidence;
• the context relevance of the clinical question at hand (is the question still relevant given considerations such as the burden of disease, variation in practice, or emerging care options?);
• methodological applicability of the clinical question (does the question still address PICO components of interest?);
• user interest in an update; and
• the potential impact of an update on access to health care.

To apply this tool in a real-world setting, the authors took a sample of four guidelines published by the Spanish National Health System (NHS) within the past 2-3 years and which utilized the GRADE framework. A survey was then developed in order to assess the above six items, calculate a priority ranking, and from there, decide which questions were in highest need of updating. The survey was disseminated among members of a working group comprising members of the original guideline and additional content experts. Additional factors for consideration included the volume of new evidence, the availability of resources, and the need to include new clinical questions. 

Through this process, a total of 16 (15%) of the 107 questions were defined as high priority for updating.  Of these, 12 were given a score higher than five for one of the individual items (specifically the item assessing an impact on patient safety), while the remaining four received an overall score higher than 30 across all six items.

In addition to the priority ranking derived from the six assessment items, the survey also assessed the usability and inter-observer reliability of the tool itself. The reliability (intra-class correlation) ranged from good in one guideline (0.87) to moderate (0.62 and 0.63) in two guidelines and poor (0.15) in one. The authors conclude that the identification and proper training of content experts to serve as appraisers remains the key challenge for the efficacious application of this tool.

Sanabria, A.J., Alonso-Coelle, P., McFarlane, E., et al. (2021). The UpPriority tool supported prioritization processes for updating clinical guideline questions. J Clin Epidemiol (in-press).

The manuscript can be accessed here.

Expert Evidence: A Framework for Using GRADE When "No" Evidence Exists

To guide the formulation of clinical recommendations, GRADE relies on the use of direct or, if necessary, indirect evidence from peer-reviewed publications as well as the gray literature. However, in some cases, no such evidence may be found even after an extensive search has been conducted. A new paper - part of the informal GRADE Notes series in the Journal of Clinical Epidemiology - relays the results of piloting an "expert evidence" approach and provides key suggestions when using it.

As opposed to simply asking the panel members of a guideline to base their recommendations off of informal opinion, the expert evidence approach systematizes this process by eliciting the extent of their experience with certain clinical scenarios through quantitative survey methods. In this example, at least 50% of the panel members were free of conflicts of interest, with various countries and specialties represented. While members were not required to base their answers off of patient charts, the authors suggest that this can be used to further increase the rigor of the survey. 

As a result of the survey, the recommendations put forward reflected a cumulative 12,000 cases of experience. Because the members felt that at least some recommendation was necessary to help guide care - where the alternative would be to provide no recommendation at all - the guideline helped to fill a gap while indicating the current lack of high-quality published evidence for several clinical questions, which may help guide the production of higher-quality evidence and recommendations in the future. Importantly, by utilizing a survey approach to facilitate the formulation of recommendations, the authors note that it avoided the pitfall of "consensus-based" approaches to guideline development which can often manifest as simply reflecting the opinions of those with the loudest voices. 

Mustafa RA, Cuello Garcia CA, Bhatt M, Riva JJ, Vesely S, Wiercioch W, ... & HJ Schünemann. (2021). How to use GRADE when there is "no" evidence? A case study of the expert evidence approach. J Clin Epidemiol, in-press. 

Manuscript available from the publisher's website here. 

RIGHT-PVG: A New Checklist for the Reporting of Patient Versions of Guidelines

 Patient versions of guidelines (PVGs) can provide crucial information about diagnoses and management options to patients in clear, plain language and can help guide shared decision-making between patients and their providers to improve the quality of care. However, the construction and reporting of PVGs is variable in terms of quality and content. Now, a new extension of the Reporting Tool for Practice Guidelines in Health Care - the RIGHT-PVG - aims to standardize the development of such documents.

Development of the RIGHT-PVG comprised 17 experts from around the world with experience in guideline development, patient communication, and epidemiology, and clinical practice. First, an initial list of items was generated from common themes in a sample of 30 PVGs. Then, four organizational guidance documents for the development of PVGs were identified and used to refine initial criteria. Two rounds of a modified Delphi consultation were used to further pare and refine checklist items from an original list of 45, with all panelist feedback anonymized. 

Final items included within the RIGHT-PVG fell under four main categories:

• Basic information: items 1-3 include the reporting of title and copyright, contact information, and a general summary of the PVG's key points.
• Background: items 4-6 include a general introduction to the topic at hand, information about the scope and target audience of the document, and a link to the original guideline off of which the PVG is based.
• Recommendations: items 7 and 8 comprise the meat of the PVG: what is the guideline recommending, for whom, and what are the potential desirable and undesirable effects of the intervention? 
• Recommendations should be easily identifiable via boxing, shading/coloring, or bold type.
• The strength of each recommendation should be included along with a transparent reporting of the certainty of the evidence behind it.
• Easy-to-understand symbols can be used to denote the differences between strong and more conditional recommendations.
• Other information: items 9-12 recommend the inclusion of suggested questions for the reader to ask their provider; a glossary of terms and abbreviations; information about how the guideline was funded; and disclosure of any relevant conflicts of interest.

Wang X., Chen Y., Akl E.A., ... and the RIGHT working group. (2021). The reporting checklist for public versions of guidelines: RIGHT-PVG. Implement Sci 6(10). 

Manuscript available from the publisher's site here. 

Help for Choosing Among Multiple Interventions Using GRADE

It is not uncommon for a health guideline to compare two or more interventions against one another. However, while sophisticated statistical approaches such as network meta-analyses allow us to compare these interventions head-to-head in terms of specified health outcomes, they do not take other important aspects of clinical decision-making into account, such as patient values and preferences, resource use, and equity considerations. A new paper from Piggott and colleagues aims to provide initial suggestions for using the GRADE evidence to decision (EtD) framework when choosing which of multiple interventions to recommend.

The authors identified a need for more direction when undertaking a multiple intervention comparison (MC) approach while working on recently released guidelines for the European Commission Initiative on Breast Cancer in which multiple screening intervals were compared against one another. Based on this experience, the group drafted a flexible yet transparency-minded framework to help guide similar efforts in the future, which was then added as a module in GRADE's official guideline development software, GRADEpro. 

The new module was pilot-tested for feasibility with several additional guidelines. The module allows the user to select and then compare multiple pairwise comparisons against one another (for instance, with one column for "Intervention 1 vs. Comparator 1" and "Intervention 2 vs. Comparator 2"). A five-star system is used to judge various components of the EtD, such as cost effectiveness, for each individual intervention and comparator, whereas a column on the right-hand side allows the user to input the relative importance of these components in decision-making.

Finally, the user can review all judgments across interventions and summatively recommend the most favorable intervention(s) overall.

Piggott T, Brozek J, Nowak A, et al. (2021). Using GRADE evidence to decision frameworks to choose from multiple interventions. J Clin Epidemiol 130:117-124.

Manuscript available from the publisher's website here.

New Guideline Participation Tool Lays Out Roles and Responsibilities for New and Returning Guideline Group Members

Guideline development groups should contain a multidisciplinary panel of experts and key stakeholders to ensure the quality, relevance, and ultimate implementation of resulting recommendations. However, there are few tools in existence to ensure the effective participation of panel members when working to draft guidelines, and preparing panel members with little to no previous experience in guideline development can be an especially daunting task. A new paper published in next month's issue of the Journal of Clinical Epidemiology aims to provide a tool to guide these efforts, with a specific focus on guideline developed using the GRADE framework.

To develop the tool, Piggott and colleagues first established a draft tool that included 61 items based on a previously published systematic review of guideline development handbooks. They then conducted a series of ten key informant interviews comprising both past and prospective guideline development group members to narrow the tool down to three major themes: selection of participants, guideline group process, and tool format. The resulting 33-item Guideline Participant Tool (GPT) was then validated in a survey of 26 guideline group members from various societies including WHO and the American Society of Hematology (ASH). The tool itself breaks the process of guideline participation into three major time windows: 

• Before (Preparations): 12 items including clarifying objectives and one's role within the group and familiarizing oneself with the guideline development methodology to be used.
• During (Meetings): 15 items including avoiding undue interruptions, adhering to the specified methodology, and referring to the PICO question at hand as a way to stay on task.
• After (Follow-up): 6 items including maintaining proper confidentiality of information discussed, reviewing meeting minutes to identify any discrepancies in a timely fashion, and assisting with the promotion, dissemination, and evaluation of the guideline as requested.

According to the authors, "Most participants found that the tool is most useful before guideline group meetings explaining what to expect at each phase. Participants thought that the tool was useful beforehand as a reference for orienting themselves to the structure of meetings, understanding the guideline development process, and what might be required of them. Respondents agreed that the tool serves as a reference for them to stay on track with the required tasks and to support structuring the process of guideline development."

The authors go on to suggest that the tool be used as required reading for all group members ahead of their participation on a panel. 

Piggott T, Baldeh T, Akl EA, et al. 2021. Supporting effective participation in health guideline development groups: The Guideline Participant Tool. J Clin Epidemiol 130:42-48.

Manuscript available from the publisher's website here. 

Practical Tips for Finding and Assessing Patient Survey Data

 An essential part of translating a body of evidence into a clinical recommendation within the GRADE framework is the consideration of patients' values and preferences. Not only should the likely treatment preferences and values placed on outcomes among the patient population be considered; if there is likely a great amount of variability within these, this may also influence the ultimate strength of recommendation.

Guideline panels and public health decision-makers may use self-reported patient survey data to better understand the range of patient values and preferences when formulating recommendations or policies. However, like all sources of evidence, patient surveys may be at risk for specific sources of bias which can ultimately affect the results. What should decision-makers look out for when applying patient survey data to a recommendation for care? In a recently published paper, Santesso and colleagues propose a practical guide for finding, interpreting, and applying patient data to better inform healthcare decision-making.

Click to enlarge.

Because 97% of published surveys have been found to use the words "survey" or "questionnaire" in the title, the authors suggest using these terms in title, abstract, and topic fields when conducting a search for relevant data. When assessing the risk of bias of a given survey, decision-makers should ask whether the population was adequately representative of the patient population in question, taking care to consider the use of random sampling and the potential impact of nonresponse. A survey should also be assessed for whether it measures the intended constructs adequately. Survey authors should report the variability around reported measures whenever possible, and these data can be used to judge the overall variability in patient values and preferences. Finally, decision-makers should take care to discern how directly the survey data applies to the patient population in question; the table of survey respondent characteristics is a useful place from which to draw judgments of directness.

Using these helpful and practical points of guidance, guideline panel members and clinical decision-makers can better inform their retrieval, critical appraisal, and application of patient survey data to important healthcare questions, ultimately resulting in more informed guidelines and policies.

Santesso N, Akl E, Bhandari M, Busse JW, Cook DJ, Greenhalgh T, Muti P, Schünemann H, and Guyatt G. (2020). A practical guide for using a survey about attitudes and behaviors to inform health care decision making. J Clin Epidemiol 128:93-100.

Manuscript available from the publisher's website here. 

Evidence Foundation Welcomes Four Scholars in First Virtual Workshop

In late October, the U.S. Grade Network held its thirteenth GRADE Guideline Development Workshop. Like any of the twelve workshops before it, there was much learning, discussion, and networking to be shared. However, unlike any workshop in the past, it was fully online.

Among the 45 attendees who participated in offices and living rooms from Brazil to Cyprus were four participants who attended the workshop free of charge as recipients of the Evidence Foundation scholarship. During a virtual Evening with the Fall 2020 Evidence Foundation Scholars, these four bright minds presented briefly on a proposal or current project designed to reduce bias in healthcare.

Dr. Stavros Antoniou, Chair of the European Association for Endoscopic Surgery Guidelines Subcommittee, discussed the tripartite Guideline Assessment Project (GAP) aimed at developing an extension of the AGREE II tool for surgical guidelines. In an exploratory analysis published earlier in 2018 (GAP I), Antoniou and colleagues assessed 67 surgical guidelines and reported that development of more than one guideline per year, the presence of a guideline committee, and the use of GRADE was associated with higher scores in AGREE II. Second, the group explored the reliability, internal consistency, and unidimensionality of the AGREE II tool when applied to surgical guidelines (GAP II). The group is now in the process of using the Delphi process to identify and finalize items for the surgical extension based on stakeholder input, pilot-testing the instrument, and assessing its validity (GAP III). Of the workshop. Dr. Antoniou noted, "participating in the GRADE Guideline Workshop as a scholar was an inspirational experience. It was fascinating to be trained by world-renowned experts, who have embraced us with true interest and conveyed their passion with quality in guideline development."

Jung Min Han, PharmD, MS, manages the development of guidelines for the American Academy of Dermatology. Her presentation reviewed her current project to update the organization's 2016 guidelines on the management of acne vulgaris using the GRADE framework. Ms. Han discussed the plan to organize two working groups, one to review and update the nine clinical questions from the previous guidelines, and the other to add additional new questions as needed. An updated search would then be run for the first set of questions to identify any newly published evidence since the original guidelines were developed; simultaneously, a novel systematic search would be conducted for the second group of questions. New recommendations would then be drafted following the GRADE methodology. Ms. Han stated, "The GRADE Workshop has trained me to confidently use GRADE in different scenarios where head-to-head data from randomized controlled trials are not available. The workshop was very well-structured with a concrete theme and a mix of lectures, small and large group discussions, meet the experts Q&A sessions, and real-world examples that challenged trainees in many ways."

Dr. Georgios Schoretsanitis of Zucker Hillside Hospital in Glen Oaks, New York presented on his work developing guidelines for therapeutic drug monitoring to optimize and tailor treatment for psychotherapeutic medications. Beginning in 2017, a series of recommendations for reference ranges for two commonly prescribed antipsychotic medications was developed, followed this year by an international joint consensus statement on blood levels to optimize antipsychotic treatment in clinical practice. "For long I have been interested in conducting systematic reviews and meta-analyses," said Dr. Schoretsanitis. "Attending the GRADE Guideline Workshop organized by the US GRADE Network gave me exactly what I was looking for: a unique chance to essentially deepen my knowledge on major methodological aspects during stimulating lectures by experts that have set the tone in the field. It was an intense experience far beyond acquiring knowledge, which I highly suggest to every methodologist."

Dr. Zeinab Hosseini, a Saskatchewan Health Research post-doctoral fellow at the University of Saskatchewan, discussed her work examining the impact of exercise interventions on osteoporosis. Because gender and sex affect the prognosis and management of the disease, guidelines that consider these differences are needed, she said. As part of her research under the advisement of Dr. Phil Chilibeck, she hopes to contribute further understanding in the field related to gender- and sex-specific considerations for exercise recommendations in patients with osteoporosis, and to help inform future guideline recommendations on this topic. "The US GRADE Network Workshop was an amazing opportunity for me as a post-doctoral fellow in health proving insight on how to think as a health researcher from early stages of research up to knowledge translation and dissemination and how to provide evidence-based recommendations to inform the public considering situations where the literature is scarce," said Dr. Hosseini. "There are top women and men scientists on the training panel who respond to questions using their experiences as member on different panels, which I think is unique."

The USGN facilitators pose for a virtual group photo with the four fall 2020 Evidence Foundation scholars. Click to enlarge.

The Evidence Foundation thanks all four scholars for attending and contributing their engagement and expertise to our 2020 fall workshop.

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Chicago, Illinois will be February 28, 2021.

U.S. Guideline-Producing Organizations Show Some Promise, Room for Improvement in their Application of GRADE

As many as one-third of guideline-producing health organizations in the United States report using the GRADE framework, but exactly how closely these organizations follow the key tenets of GRADE - such as using evidence summaries of each identified outcome to inform the overall certainty of evidence, and linking this certainty to a strength of recommendation - is a matter of debate.

In study by Dixon and colleagues published earlier this year in the Journal of Clinical Epidemiology, the authors set out to evaluate the use of GRADE in U.S.-based guidelines published between 2011 and 2018 and available in the National Guidelines Clearinghouse. Assessing up to three of the most recent guidelines from each of 135 identified U.S.-based organizations, the authors used several criteria to examine how closely each of the 67 resulting guidelines adhered to core GRADE concepts, including:

• defining the certainty of evidence,
• explicitly considering the GRADE domains when assessing the certainty of evidence, and
• consistently defining the strength of resulting recommendations as strong or weak/conditional.

While most (89.6%) defined the certainty of evidence in a matter consistent with GRADE, only 10.4% explicitly reported examining certainty through all 8 GRADE criteria. Only 13.4% of guidelines assessing the certainty of evidence sourced from non-randomized trials reported assessing the potential reasons to upgrade the certainty of evidence (i.e., large magnitude of effect, dose-response gradient, and residual confounding). Finally, only about half (53.7%) provided an evidence profile or summary of findings table describing the assessments, and while reporting of the certainty of evidence and the balance between desirable and undesirable effects was most common (100% and 97%, respectively), explicit consideration of resource use and patients' values and preferences were also fairly common (73.1% and 77.6%, respectively) .The use of GRADE in line with the authors' established criteria appeared to grow somewhat more frequent over time, indicating a general trend toward proper use of GRADE.

Figure from Dixon et al. shows the relative reporting frequency of the various GRADE criteria for assessing certainty of evidence in years 2011-14 versus 2015-18, suggesting a trend for improved reporting over time. Click to enlarge.

The authors conclude that continued training of guideline developers and dissemination of education on the appropriate application of GRADE should further improve adherence, including the explicit consideration of all eight domains for assessing the certainty of evidence and of all aspects that inform the translation of this evidence into clinical recommendations.

Dixon C, Dixon PE, Sultan S, Mustafa R, Morgan RL, Murad MH, Falck-Ytter Y, and Dahm P. (2020). Guideline developers in the United States were inconsistent in applying criteria for appropriate Grading of Recommendations, Assessment, Development and Evaluation use. J Clin Epidemiol 124:193-199.

Manuscript available at the publisher's website here.

Research Revisited: "Quality of Evidence is a Key Determinant for Making a Strong GRADE Guidelines Recommendation" (2015)

This month in 2015, Djulbegovic and colleagues published a paper that examined the impact of quality of evidence, balance between benefits and harms, patient values and preferences, and resource use (the four GRADE factors) on the strength of resulting clinical recommendations.

The four major GRADE factors that drive clinical recommendations 

The authors circulated a survey among 18 members of a guideline panel of the American Association of Blood Banking (AABB) who had recently convened to develop guidelines for the use of prophylactic versus therapeutic platelet transfusion in patients with thrombocytopenia. Using the panel members’ assessments of the GRADE factors with regards to the evidence that had been presented and their resulting recommendations for strong or weak recommendations, a logistic regression was conducted in order to examine the relative impact of each of the four GRADE factors.

The guideline panel had reviewed the evidence for ten key questions. Overall, the consistency of judgments across panel members was good (Cronbach’s alpha = 0.86). Those questions with a high quality of evidence were 4.5 times more likely to result in a strong recommendation (p < 0001), whereas none of the three remaining GRADE factors were significantly associated with the strength of the resulting recommendations. Moreover, the model suggested that in cases where the quality of evidence was high, there was a 90% chance of the resulting recommendation being strong; when the quality of evidence was very low, this chance dropped to 10%.

The figure from Djulbegovic shows the associations between increasing quality of evidence and strength of resulting recommendations

The authors concluded that the quality of evidence was far and away the most important contributing factor to the resulting strength of recommendations, at least within the studied guideline panel. However, it’s important to note that patient values and preferences, the balance between benefits and harms, and issues of resource use should all be involved in the process of moving from evidence to decisions, but the relative nebulousness around these considerations and a lack of determined structures for eliciting the data they require (such as the use of a patient panel or survey) likely make them less impactful on the overall strength of recommendation. 

The Evidence-to-Decision framework, which makes these additional considerations more explicit in the formulation of recommendations, was introduced just a year later. It’s worth wondering whether the relative impact of the remaining three GRADE factors has changed since the introduction and adoption of this framework – perhaps presenting an opportunity to revisit this research.

Djulbegovic, B., Kumar, A., Kaufman, R.M., Tobian, A., and Guyatt, G.H. Quality of evidence is a key determinant for making a strong GRADE guidelines recommendation. J Clin Epidemiol 68(7): 727-732. 

Manuscript is available at the publisher's website here. 

Research Revisited: 2014’s “Guidelines 2.0: Systematic Development of a Comprehensive Checklist for a Successful Guideline Enterprise”

While several checklists for the development and appraisal of specific guidelines had been developed by 2014, there had yet to be published a thorough and systematic resource for organizations to inform the actual day-to-day operations of a guideline development program. Noticing this need, Schünemann and colleagues pooled their professional experiences and contacts in the field in addition to conducting a systematic search for self-styled “guidelines for guidelines” and other guideline development handbooks, manuals, and protocols. The reviewers, in duplicate, extracted the key stages and processes of guideline development from each of these documents, compiling them together.

The result was the G-I-N/McMaster Guideline Development Checklist: an 18-topic, 146-item soup-to-nuts comprehensive manual spanning each part and process of a guideline development program, from budgeting and planning for a program to the development of actual guidelines to their dissemination, implementation, evaluation, and updating.

An overview of the steps and parties involved in the G-I-N/McMaster guideline development checklist. Click to enlarge.

The checklist also provides hyperlinks to tried-and-true online resources for many of these aspects, such as tips for funding a guideline program, tools for project management, topic selection criteria, and guides for patient and caregiver representatives.

Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, Ventresca M et al. Guidelines 2.0: Systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ 186(3): E123-E142.

Manuscript available for free here.

Don't Sell Your Guidelines Short - Remember to Report! (Part 2)

Part II: The Most Common Errors in Guideline Reporting (and Their Easy Fixes)

In Part I, we discussed the importance of reporting crucial details about the development process of a guideline and the various tools and checklists used to assess the quality of guidelines through what is reported.

In this post, we’ll provide a list of items that guidelines commonly neglect to report but which can easily and instantly improve the chances of a positive quality appraisal of a guideline. Remember: items that don’t have to do with the recommendations directly can always be described in detail elsewhere, such as in a supplemental document or a web site. Just make sure all materials are referenced in the guideline text so that its users – and those who may be assessing the guideline for research purposes – can easily find the relevant information.

1.     The target audience. Be explicit about who the guideline is and isn’t targeted toward. Rather than “healthcare professionals,” specifically list specialties and/or settings when appropriate (e,g., neurologists, primary care providers in low-income communities).

2.     The clinical questions. All good answers must begin with a well-formulated question, but it is surprisingly common for guidelines to leave out an explicit description of the key questions driving the guideline. Why was this guideline created? What questions are the recommendations addressing? Simply listing all questions in PICO (Population, Intervention, Comparison, Outcome) format at the beginning of the text and even as subheaders throughout the document will effectively guide readers through the text and provide helpful context for the appropriate application of recommendations.

3.     The patient voice.  The use of patient/consumer representation in the planning of a guideline and in drafting the resulting recommendations appears on many assessment tools and standard sets. If patients or consumers are involved in the development of a guideline at any part of the process, make sure to describe how and when – including whether educational strategies to increase their effective participation where used. Plans for dissemination of recommendations to the public beyond healthcare providers also apply here.

4.     The external review and public comment process. How were external reviewers selected? Were their identities kept confidential from the guideline authors? Did the guideline panel keep a formal documentation of all reviewers’ comments and respond to each one specifically? Lastly, was the guideline available for public comment, and how long was it available? The addition of these fine details is relatively easy but can drastically improve the assessment of a guideline in this domain.

5.     Implementation considerations. A perfect guideline is worth nothing if it isn’t followed. What are some potential barriers for appropriate implementation of the guideline of which its users should be aware? Have any tools been developed to improve the chances of successful implementation? How might resource availability affect implementation, and how should implementation be monitored?

6.     Conflicts of interest. This crucial aspect of guideline development goes further than simply listing interest disclosures of the authors. If any potential conflicts exist, describe how they were managed throughout the process. If the guideline has been funded by external sources, it’s important to note this and to describe the terms of the funding.

7.     Plans for update. Along with indicating the date of the relevant review of evidence, describe any plans and procedures for surveillance of the literature and updating of the guidelines. In addition, provide an expiration date for the current guideline.

Whether being read by a provider with the intention of implementing its recommendations or by a researcher assessing its quality, a guideline is only as good as how well it is written. Including as much information as possible, whether in the text of the guideline itself or in a referenced external source, will improve the chances of a fair appraisal of a guideline’s quality – and, more importantly, its successful implementation.

Rapid Guidelines in GRADE Pt. III: A checklist for rigorously rapid recommendations

In recent posts, we have introduced the concept of rapid recommendations as well as how developers of these recommendations at the World Health Organization (WHO) perceive facilitators and barriers to this process. This information was gathered as part of a published series on rapid guidance in 2018.

In the final part of the series, Morgan and colleagues propose an extension of the G-I-N/McMaster Checklist for Guideline Development aimed at those producing rapid guidelines. Comprising 21 discrete guideline principles that align with the original Guideline Development Checklist, the checklist is a tool for developers to take stock of the resources available for their rapid guideline development goals and identify areas in need of improvement. Important considerations include:

·      Make use of virtual meetings (Principle 7) and pre-meeting voting (Principle 15) to expedite the drafting of recommendations.

·      If possible, limit guideline panel composition to those not reporting financial COIs, or if not possible to provide necessary topic expertise, transparently declare any modifications of existing COI policies for the topic at hand (Principle 9).

·      Limiting the number of PICOs (Principle 10) and limiting the assessment of outcomes to only those deemed critical (Principle 11).

·      Consider ways to facilitate systematic review stage, such as updating existing reviews, developing rapid reviews, or tailoring search criteria to a smaller scope (Principle 13)

·      Arrange for external reviewers early on in the process so that they are quickly deployed when a draft is available for review (Principle 18).

The full checklist extension for rapid guidelines can be viewed here.

Morgan, R.L., Florez, I., Falavigna, M. et al. Development of rapid guidelines: 3. GIN-McMaster Guideline Development Checklist extension for rapid recommendations. Health Res Policy Sys 16, 63 (2018). https://doi.org/10.1186/s12961-018-0330-0

Manuscript available at the publisher's website here.

Rapid Guidelines in GRADE Pt. I: Needed Advice when Time is of the Essence

While most clinical practice guidelines take 2-3 years to develop and publish, the emergence of a public health crisis or urgent humanitarian need requires the dissemination of evidence-based guidance in a more rapid manner. To this effect, several national- and international-level guideline-producing organizations, such as the Centers for Disease Control (CDC) and the World Health Organization (WHO), have developed processes for the development of evidence-based guidance for these more urgent situations.

WHO’s 2006 recommendations for the pharmacological management of avian influenza in humans is one example of a rapidly developed guideline. Of current relevance, WHO has recently published interim guidance on the management of severe acute respiratory infection when novel coronavirus is suspected, and the UK’s National Institute for Health and Care Excellence (NICE) has also developed interim guidelines for the treatment of COVID-19 in patients receiving critical care, kidney dialysis, and systemic anticancer therapy. Because this matter is rapidly evolving and advice is needed immediately, the protocols used by NICE, WHO and other organizations are different than it would be for less urgent topics.

Can rapid guidelines use GRADE?

In short, yes. Recommendations can be made based on the transparent grading and reporting of the certainty of evidence that lie at the heart of GRADE, whether this is over the timeframe of hours, days, weeks, or months. The key word here is transparent: no matter the speed of development, recommendations should always be couched within the terms of the certainty of evidence behind them, and judgments of the evidence should be clearly presented. In a 2016 paper on the use of GRADE to respond to health questions with different levels of urgency, Thayer and Schünemann provide terms for the various speeds of response, and considerations for recommendations therein:

• Ultra-short emergency response: 1 or more hours

• Urgent response: 1-3 weeks

• Rapid response: 1-3 months

• Routine response: More than three months

Recommendations can still be formed based on the certainty of the evidence that's available, whatever that evidence may be. While systematic reviews of all available evidence are a foundational aspect of non-urgent guidelines, evidence in the form of narrative syntheses, modeling, or late-breaking data from the field can be used when time is short and systematically compiled data are sparse. Regardless of the source, the domains of GRADE still allow for evidence to be appraised and to guide the resulting direction and strength of recommendations.

Stay tuned for Pt. II coming soon, where we'll take a closer look at organizations that have developed rapid recommendations in response to time-sensitive public health issues.

For a checklist to guide the development of rapid recommendations, see the G-I-N/McMaster checklist.

For more information about appraising the certainty of evidence in the lack of meta-analyzed data, see this paper.

Thayer KA & Schünemann H. Using GRADE to Respond to Health Question With Different Levels of Urgency. Environment international. 2016 July-August: 585-589.

Manuscript available at the publisher's website here.