Spring 2019 - Scholarship Recipients

Contributed by Madelin Siedler, 2018/2019 U.S. GRADE Network Research Fellow

Recently, we held the Tenth GRADE Guideline Development Workshop in Denver, Colorado. This workshop was one of the largest groups to date, with 51 participants traveling to the Mile-High City from as far away as Poland and Korea. During the workshop, participants focused on learning and applying the GRADE approach for diagnostic test accuracy. 

Two participants attended as recipients of the scholarship program funded by the U.S. GRADE Network and Evidence Foundation. This scholarship covers the cost of registration for workshop attendees who are newer to GRADE and have never attended a formal GRADE workshop. Scholars Janice Tufte and Dr. Irbaz bin Riaz presented on their innovative ideas for improving the development, implementation, or dissemination of guidelines with the aim of reducing bias in healthcare recommendations.

Scholarship recipients: Dr. Irbaz bin Riaz (L) and Ms. Janice Tufte (R), 

with scholarship coordinator, Dr. Shahnaz Sultan

Tufte, an independent consultant who leads patient-public partnership initiatives, presented on the unique opportunities of using patient partners during the development of GRADE guidelines. Patient partners are representatives of the patient population whom the guideline aims to serve. As part of a guideline panel, they offer fresh perspectives, ground the guideline development process with lived experience, and help the panel to identify and address differences in priorities among stakeholders.

Throughout her presentation, Tufte provided ways to improve how patient partners are involved in the guideline process, such as creating one-pagers and glossaries that cover the basics of GRADE methodology and inquiring beforehand about specific accommodations that might be needed in order to enhance the patient’s participation in the panel. “It was an honor to attend the GRADE Workshop in Denver as a Patient Partner Scholar,” said Tufte. “I felt like I was treated like a colleague where we were all learning together how to use GRADE tools to share best evidence within our individual systems and guidelines work.”

Dr. bin Riaz, an oncologist at Mayo Clinic, presented on a framework for developing living systematic reviews and guidelines to inform clinical decision-making, especially in topic areas undergoing rapid change. It can take several years for a systematic review and resulting clinical recommendations to be developed, Dr. bin Riaz explained. In the meantime, new drug approvals or indications, changes in drug labeling, or new information about potential risks and benefits of a treatment option can arise. As opposed to traditional, static documents, living systematic reviews and guidelines are continually updated as new evidence or important decision-making information comes to light. The ultimate goal of such an approach is to facilitate a more timely translation of medical knowledge into clinical practice, allowing patients and their providers to come to decisions informed by the totality of current evidence.

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Orlando, Florida will be July 1, 2019.

Research Shorts: Borrowing of strength from indirect evidence

Contributed by M. Hassan Murad, M.D.

Network meta-analysis (NMA) is supposed to increase precision (because it includes more studies in the analysis); however, this is not always the case. This empirical study evaluated 915 possible treatment comparisons. The study used the recently proposed borrowing of strength statistic, which quantifies the percentage reduction in the uncertainty of the effect estimate when adding indirect evidence to an NMA

When only one study contributed direct evidence, NMAs resulted in reduced precision and no appreciable improvements in precision in 57.5% and 12.7% comparisons, respectively. When at least two studies contributed direct evidence, NMAs provided increased precision in 66.4% comparisons. The bottom-line is that, in sparse networks (i.e., networks that mostly consist of indirect evidence), NMA may not improve precision as much as stakeholders want and expect.

Reference: Lin L, Xing A, Kofler MJ, Murad MH. Borrowing of strength from indirect evidence in 40 network meta-analyses. Journal of clinical epidemiology. 2018 Oct 17.

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

Contributed by Madelin Siedler, 2018/2019 U.S. GRADE Network Research Fellow

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

One of the benefits of the GRADE approach is that it provides a framework for the development of evidence-based recommendations that are clear and actionable for practicing clinicians even when only lower-quality evidence is available. However, in some particular instances, caution is warranted when developing a recommendation based on low-quality evidence or inference. “Good practice statements” are one such instance. 

The term “good practice statement” is sometimes interchanged for “motherhood statement”.” In either case, the practice being recommended is usually something that is already commonly accepted as beneficial or practical advice. It could even be seen as irrefutably “good” as motherhood and apple pie (hence the term). The nature of these types of statements is such that the action is seen as so obviously beneficial that it would be unduly onerous to conduct a review to demonstrate its efficacy.

An example of a good practice statement is the first recommendation from the American Gastroenterological Association (AGA)’s 2015 guideline on the management of asymptomatic pancreatic cysts, which reads, “The AGA recommends that before starting any pancreatic cyst surveillance program, patients should have a clear understanding of programmatic risks and benefits” (Vege et al., 2015).

How to Spot a Good Practice Statement
An easy way to identify a good practice statement is to restate the recommendation as its inverse: for instance, “patients should not have a clear understanding of programmatic risks and benefits.” If this “unstated alternative” is absurd or clearly does not conform to ethical norms, the original statement is likely a good practice statement (Guyatt et al., 2016).

The Problem with Good Practice Statements
The GRADE Working Group recommends that good practice statements be used sparingly, if at all. Because good practice guidance is typically based on several linked sources of indirect evidence, there is no way to tell whether the benefits of the proposed recommended action are as truly obvious or incontestable as they seem. And if they are (if the inverse of the proposed recommendation would be absurd or unethical) then they are likely unwarranted and can dilute the strength of the guideline as a whole. 

Sometimes, good practice statements may even appear as graded recommendations in a guideline (a decision that’s not recommended by the GRADE working group for the reasons above). In this case, the guideline authors may be tempted to make a strong recommendation based on low-quality evidence, which should ideally be a rare occurrence and based on well-defined criteria (Guyatt et al., 2015). 

Practical Advice for Dealing with Good Practice Statements
The GRADE Working Group recommends using the following checklist to determine whether a good practice statement is warranted:

1. Is the statement clear and actionable? 
2. Is the message really necessary in regards to actual health practice? 
3. After consideration of all relevant health outcomes and potential downstream consequences, will implementing the good practice statement result in large net positive consequences? 
4. Is collecting and summarizing the evidence a poor use of a guideline panel’s limited time and energy? 
5. Is there a well-documented clear and explicit rationale connecting the indirect evidence?

If the answer is 'yes' to all five questions, a good practice statement may be warranted for inclusion in a guideline document. When done correctly, good practice advice statements should appear as ungraded recommendations, meaning no formal rating of quality of evidence or strength of recommendation should be given (Guyatt et al., 2015).

However, many potential good practice statements will be eliminated through the use of this checklist. For instance, careful consideration of Question #3 could lead the panel to realize that the assumed net positive of a specific action may not be so obvious after all. In this case, the guideline panel should consider whether a thorough review of the evidence should be conducted and formal grading methods applied.

Research Shorts: When continuous outcomes are measured using different scales

Contributed by M. Hassan Murad, MD

Outcomes of great importance to patients, such as quality of life and severity of anxiety or depression, are often measured using different scales. When an outcome is measured using several scales across trials, it requires standardization to be pooled in a meta-analysis.

Common methods of standardization include using the standardized mean difference (SMD), converting continuous data to binary relative and absolute association measures, the minimally important difference (MID), the ratio of means, and transforming standardized effects back to original scales. The underlying assumption in all these methods is that the different scales measure the same construct. This paper, in BMJ, describes these methods and suggests approaches for interpretation.


Reference: Murad Mohammad Hassan, Wang Zhen, Chu Haitao, Lin Lifeng. When continuous outcomes are measured using different scales: guide for meta-analysis and interpretation BMJ 2019; 364 :k4817. https://www.bmj.com/content/364/bmj.k4817 

Fall 2018 Scholarship Recipients

Contributed by Madelin Siedler, 2018/2019 U.S GRADE Network Research Fellow

We were pleased to support the participation of three research scholars at our recent GRADE guideline development workshop held in Silver Spring, Maryland, October 17-19, 2018. By providing complementary registration, we hope that these scholars increased their understanding and application of the GRADE approach. As part of the U.S. GRADE Network/Evidence Foundation scholarship, recipients presented to fellow workshop participants about their interests and current endeavors in the field of guideline development.

Scholarship recipients: Oilvia Magwood, Mohamad Kalot, and Mohammed Alkhatib

A few details about our Fall 2018 scholarship recipients:

Mohamad Kalot, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in improving quality and decreasing disparities in healthcare through the development of evidence-based guidelines. Dr. Kalot presented on his current work in the development of guidelines for the management of rare diseases. Dr. Kalot explained how the development of such guidelines presents a unique challenge in that there is often a dearth of research on these populations, which can affect the directness of evidence among other factors.

“Since an important part of my conducted reviews and research deals with developing guidelines for rare diseases, I would ultimately face specific challenges in the process of assessing the certainty of evidence in my work, and I’m interested in exploring innovative methods to deal with these challenges,” said Kalot. “The GRADE workshop in Silver Spring didn’t only help me find solutions for my challenges, it made me see the research methodology world from a different, deeper and more practical perspective - especially with the very helpful tools [RevMan and GRADEpro Guideline Development Tool] that we learned about in the small groups sessions and the discussions about rating up and rating down the quality of evidence.”

Olivia Magwood, MPH, attended from the Buyere Research Institute in Ottawa, Ontario, where she has participated in the development of three national and international guidelines. Ms. Magwood presented on the development of the FACE (Feasibility, Acceptability, Cost, and Equity) Framework Stakeholder Survey. The FACE Stakeholder Survey aims to identify cognitive bias that may influence guideline development as well as inform the quality and impact and improve the uptake of evidence-based recommendations in various stakeholder groups.

“For me, the GRADE workshop highlighted the importance of building my network and having the right people on your team throughout the guideline development process,” explained Magwood. “This includes involving methodologists early and throughout guideline development, as well as emphasizing patient perspectives, especially while deciding on patient-important outcomes. Guideline development is truly a collaborative process that benefits from multi-stakeholder engagement.”

Mohammed Alkhatib, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in examining and refining the adoption, adaptation and de novo development (or “adolopment,” for short) of guidelines to be used in lower-resource settings. Dr. Alkhatib presented on the importance of this approach, as well as the unique challenges of tailoring the development of clinical recommendations to settings with limited resources, such as developing nations and areas of conflict.

“It was an exceptional experience for me to present my proposal in front of high-level scholars of guideline development and to listen to their positive feedback,” said Alkhatib. “[The] GRADE workshop was very helpful for me at the level of interpretation of [systematic reviews] and how to judge strengths and weaknesses using GRADEpro.”

**If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Denver, Colorado will be January 1, 2019.

Research Shorts: Surrogate endpoints using the example of hepatitis C virus

Contributed by Claudia Dobler, MD, PhD

2018 U.S. GRADE Workshop Scholarship Recipient

Surrogate endpoints (for example laboratory or imaging results) are commonly used in clinical trials, as they require less participants and can be done in a shorter period of time compared to trials that use clinically important outcome measures such as mortality. When evidence for an intervention is almost exclusively based on trials that used surrogate outcomes, it is challenging for decision makers to determine the appropriateness of the use and reimbursement of an intervention. The common tendency in evidence-based medicine is to view results based on surrogate endpoints as less certain than results based on long term, final patient-important outcomes. The authors of this paper use the contemporary and highly debated example of the surrogate endpoint ‘sustained viral response’ (i.e., viral eradication considered to represent successful treatment) in patients treated for chronic hepatitis C virus infection to demonstrate how the validity of a surrogate endpoint can be critically appraised to assess the trustworthiness of the evidence and the implications for decision-making. They outline how the GRADE system for determining the certainty in the evidence can be used in situations where decisions for clinical practice and health policy have to be based on evidence that mainly comes from trials with indirect outcome measures.

Beyond assessing the quality of the evidence, potential benefits and harms of the intervention need to be weighed against each other and factors such as patient values, impact on healthcare equity, acceptability by patients and feasibility of the intervention need to be considered. The authors conclude that considering all these factors, a conditional recommendation for direct acting antiviral agents to treat chronic hepatitis C virus infection may be appropriate.


Reference: Dobler CC, Morgan RL, Falck-Ytter Y, Montori VM, Murad MH. Assessing the validity of surrogate endpoints in the context of a controversy about the measurement of effectiveness of hepatitis C virus treatment. BMJ evidence-based medicine 2018: 23(2): 50-53. https://ebm.bmj.com/content/22/6/199

Research Shorts: Treatment Burden

Contribution from Claudia Dobler, MD, PhD

2018 U.S. GRADE Workshop Scholarship Recipient

Treatment burden is the work that patients need to do to implement treatments and its effect on patient functioning and well-being. The work that patients do to manage their health often remains invisible to health professionals, who might therefore overestimate patient’s capacity to take on more interventions. Treatment burden includes filling prescriptions, taking medications, attending medical appointments, monitoring their health, making lifestyle or behavioral changes, managing medical equipment or devices, and having to deal with insurance companies and the financial burden of health care. The evidence to decision framework developed by the GRADE Working Group addresses some parts of the construct of burden of treatment in the domains of acceptability and feasibility, but burden of treatment is currently not included as a discrete construct. The authors of this paper explore the idea of including information on treatment burden associated with interventions in clinical practice guidelines. Information on treatment burden would enable patients to make informed decisions about treatments. High-quality methods for assessing treatment burden need to be identified and meaningful ways of adding this information to clinical practice guidelines need to be explored. In the paper, practical examples are outlined of how treatment burden could be included in guidelines for chronic obstructive pulmonary disease, type 2 diabetes, and chronic kidney disease. Making treatment burden explicit for adhering to a special diet for chronic kidney disease, for example, could include information on foods that should be avoided, extra time spent on grocery shopping and cooking, challenges when eating out, the typical number of consultations with a dietitian, transport requirements to see a dietitian, time spent documenting the diet, and required intensity of blood tests to monitor electrolytes.




Clinicians should discuss values and preferences with patients, especially for treatments associated with a high workload. This would help clinicians to understand how individual patients prioritize different outcomes and treatment burden and would help patients to understand what components of their treatment are the essentials, even on a bad day, so that they can prioritize treatments appropriately.


Dobler CC, Harb N, Maguire CA, Armour CL, Coleman C, Murad MH. Treatment burden should be included in clinical practice guidelines. BMJ 2018: 363. https://www.bmj.com/content/363/bmj.k4065