Fall 2025 Scholars Propose Innovative Projects in Cancer Care and Global GRADE Dissemination

Recently, the U.S. GRADE Network and Evidence Foundation had the privilege of welcoming three new scholars to the fall 2025 virtual GRADE Guideline Development Workshop. Scholars hailed from around the world, from the U.S. to Pakistan and Germany. 

Alexander Brooks, a Ph.D. candidate in exercise science at the University of South Carolina, presented a systematic review project in development to assess the long-term effectiveness of exercise interventions in cancer survivors. "Despite the growing number of clinical trials and guidelines in this area," says Brooks, "few systematic reviews have applied structured approaches to assess the certainty of the evidence underpinning these collective recommendations." Brooks' project directly addresses this gap in the literature while examining the trajectory of health and fitness markers in this population in the short- and long-term period after cancer treatment.

Attending the GRADE workshop "strengthened my ability to critically evaluate research and better understand how evidence is synthesized to inform recommendations," said Brooks. "The hands-on exercises were particularly valuable, and I found the workshop directly applicable to my own work."

Dr. Anna-Sophie Strauss, a resident physician at University Hospital Ulm in Germany and a Cochrane Urology review author, discussed her current review of the diagnostic test accuracy of prostate-specific membrane antigen (PSMA) imaging for the staging of primary prostate cancer. The aim of this review is to inform evidence-based guidelines for this testing strategy, ultimately "supporting more informed and unbiased decision-making in urological healthcare."

"Taking part in the GRADE Guideline Development Workshop improved my understanding of how structured evidence appraisal can be used to create transparent, patient-centered recommendations," said Strauss. "I also appreciated the opportunity to exchange global perspectives, which demonstrated how rigorous methods such as GRADE can be adapted to different clinical and health system contexts." 

Finally, Dr. Muhammad Tayyab Qureshi presented from Pakistan. A House Officer in the Pakistan Air Force Hospital Islamabad, Dr. Qureshi discussed the need for the dissemination of locally relevant guidelines in Lower- and Middle-Income Countries such as Pakistan. These efforts, he believes, will strengthen "health systems through practical, evidence-informed solutions that lead to measurable improvements in health outcomes and resource use."

The training provided in the three-day GRADE workshop will allow the scholars to apply the GRADE framework to the certainty and evidence assessment and formulation of recommendations within their specific projects. 

Interested in becoming a scholar? Applications to join us for the spring 2026 workshop in Kansas City, Missouri, May 18-20, 2026, close Saturday, January 31. See application details at evidencefoundation.org/scholarships.html.

GRADE Guideline Development Workshop - now with INGUIDE!

We at the U.S. GRADE Network are excited to announce that, for the first time ever, we have partnered with INGUIDE, a credentialing system for guideline developers, to offer free access to their Level 1 online training to participants of our Guideline Development Workshop and Guideline Fundamentals for Panel Members and Organization Staff pre-workshop course. 

This announcement comes just in time for our next workshop, held in Austin, Texas May 17-19, for which there are still spots available at our workshop website.

The INGUIDE program, the result of a collaboration between the Guidelines International Network (G-I-N) and McMaster University, provides certification of individuals involved in guideline development on four successive levels, from a Guideline Panel Member to becoming a Guideline Development Credentialing Instructor themselves. 

To obtain access to the INGUIDE Level 1 course, simply register for and attend our 3-day GRADE guideline development workshop in addition to the pre-workshop course on Guideline Fundamentals for Panel Members and Organization Staff. This 3-hour pre-workshop course will cover all of the necessary information about the practical aspects of guideline development, including:

• the importance of trustworthy guidelines
• checklists for the development of guidelines
• the structure and roles of a guideline panel
• the process of grading evidence and formulating recommendations

The main 3-day workshop will cover important aspects of assessing risk of bias, rating the certainty of evidence, and considering other important factors when using GRADE to develop a guideline.

Our pre-workshop course combined with our main three-day in-person course will provide all of the training required for a breeze-through experience online. You will receive free access to the ~2-3 hour online course, comprising four modules, that you can complete at your convenience after the workshop. Upon completion, you will be officially certified as an INGUIDE Level 1 Guideline Panel Member.

Why We Can't Wait for Our Austin Workshop

The U.S. GRADE Network's 18th workshop is just five weeks away! With a focus on assessing certainty of evidence informed by non-randomized studies, this workshop will take place in Austin, Texas - the home of Austin City Limits, the famous Bat Bridge, and some next-level barbecue joints.

We asked some of the workshop facilitators about what they're looking forward to about the trip (besides the chance to talk about GRADE and evidence-based medicine for three days straight!) Here's what they said...

Limited space remains for the workshop. Learn more and register and www.gradeconf.org. 

New Study Sheds Light on the Perks of Copublication of Cochrane Systematic Reviews

Since 1994, Cochrane has allowed the publication of certain systematic reviews to extend beyond the eponymous Database of Systematic Reviews and into the pages of a medical specialty journal with the hopes of increasing dissemination. Often, these copublished reviews will include an abridged version of the Cochrane review as well as commentary or other additional features explaining the review and its findings.

A new retrospective cohort study published in this month's issue of the Journal of Clinical Epidemiology highlights the benefits of such an approach. In brief, Zhu and colleagues investigated the rate of citation of Cochrane systematic reviews that had been copublished in a second journal versus those that were published only in the Database. Using a ratio of 2:1, the authors matched two randomly selected noncopublished reviews for each copublished review that came up in an indexed journal that had copublication agreements with Cochrane.

Out of the resulting sample of 101 copublished and 202 noncopublished reviews, the median number of citations over the first five years since publication was higher for reviews that had been copublished (71 versus 32.5, or approximately 118% higher in the copublished reviews). The median as higher for copublished reviews for the first, second, third, and fifth years specifically as well. In 19% of journals, the copublication of a review led to a subsequent increase in impact factor over the following year; this was true for 27.3% of journals during the second year. There was no clear trend over time for the rate of copublication across journals, though the total number of Cochrane reviews published during the same period generally increased. 

Zhu, L., Zhang, Y., Yang, R., et al. (2022). Copublication improved the dissemination of Cochrane reviews and benefited copublishing journals: A retrospective cohort study. J Clin Epidemiol 149:110-117. Manuscript available at publisher's website here. 

Spring 2022 Scholars Discuss Developments in Diagnostic and Environmental Health Evidence

The USGN's 16th GRADE Guideline Development Workshop, held in Chicago, was the first to be held in-person since March of 2020. In classic USGN style, participants enjoyed vibrant conversation, hours of learning, and delicious yogurt parfaits and strong coffee during morning breaks.

Two participants joined the fun and learning as part of the Evidence Foundation scholarship program, presenting to fellow attendees about their current projects related to evidence synthesis and guideline development. 

Spring 2022 Evidence Foundation scholars Kapeena Sivakumaran and Ibrahim El Mikati, center, pose for a photo between sessions in Chicago with the U.S. GRADE Network faculty (from left to right: Reem Mustafa, Philipp Dahm, Shahnaz Sultan, Yngve Falck-Ytter, Rebecca Morgan, and Hassan Murad).

Ibrahim El Mikati, a post-doctoral research fellow in the Outcomes and Implementation Research Unit at the University of Kansas Medical center, discussed his project helping to develop guidance for judging imprecision in diagnostic evidence. This approach will utilize thresholds for confidence intervals and will also introduce the concept of optimal information sizes for assessing imprecision in the context of diagnostic guidelines.

One thing that the GRADE workshop has helped me appreciate is transparency," said Ibrahim. "Having a transparent explanation of judgments provides users with trustworthy guidelines."

Kapeena Sivakumaran is currently leading two systematic reviews for Health Canada related to the impact of noise exposure and sleep disturbance on health outcomes. Challenges of these projects include a focus on short-term outcomes in the relevant literature as well as the need to incorporate multiple evidence streams, such as mechanistic data that can be interpreted in conjunction with observational evidence. 

“The workshop provided me with valuable insight into guideline development and using the GRADE approach to assess the evidence," said Kapeena. "One new thing I learned from the workshop was how automation and [artificial intelligence] can be integrated into the process of living systematic reviews to support guideline development.”

Note: applications for scholarships to attend our upcoming systematic review and guideline development workshops, held virtually, close August 12th and September 30th, 2022, respectively. See application details here.

2021 Reflections Before a Big Year Ahead

 From all of us at the U.S. GRADE Network, thanks for making 2021 a year to remember.

• We hosted our 14th and 15th workshops virtually - meaning we had the joy of talking GRADE with 95 attendees from many corners of the world, including Spain, Argentina, Pakistan, and Peru.
• We co-published an international needs assessment of guideline-producing organizations in collaboration with the Guidelines International Network.
• We announced our first ever two-day comprehensive systematic review workshop scheduled to be held virtually this coming February 1-2. Learn more and register here.
• And we continue to look forward to hosting our 16th guideline development workshop, once again live and in person, from June 29-July 1, 2022, in Chicago, Illinois. Registration now open.

GRADE faculty take a moment to pose for a picture with some attendees of the fall 2021 virtual guideline development workshop.

Stay tuned for more.

To stay up-to-date on all future news, events, and research round-ups from us, make sure to follow us on Facebook, twitter, and LinkedIn.

Thanks again for an excellent year, and our best and brightest wishes for a brilliant 2022.

Spring 2020 - Scholarship Recipients

Earlier this month, the U.S. GRADE Network welcomed 51 participants to the Twelfth GRADE Guideline Workshop in sunny Phoenix, Arizona. Two participants, Giulia Lane and Hayley Dunnack, were awarded a scholarship from the Evidence Foundation that covered their workshop registration fee. As the two latest additions in a line of over 40 Foundation scholars since 2014, Lane and Dunnack presented to their fellow participants about their projects related to improving the use of evidence-based decision-making in healthcare.

Hayley Dunnack, left, and Giulia Lane, right, were the recipients of the Evidence Foundation scholarship for the spring 2020 GRADE workshop held in Phoenix, AZ

Hayley Dunnack: Opportunities and Challenges in Guidelines for Nursing Practice

Hayley Dunnack, BSN, CMS-RN, OCN is a PhD candidate at the University of Connecticut School of Nursing. Dunnack presented on her work with the Oncology Nursing Society (ONS) to develop guidelines on symptom management for patients with cancer. Currently, ONS has several projects underway to develop guidelines on symptom management based on systematic reviews of the evidence and using GRADE guideline methodology. A current challenge in the ONS’ guideline development relates to the development of guidelines on topics for which randomized studies are sparse or difficult to conduct, such as the management of hypersensitivity reactions or the use of vascular access devices.  Dunnack’s project aims to examine the use of non-randomized and “real-world” evidence to inform the development of rigorous guidelines based on systematic reviews and using the GRADE process.

“I had an incredible experience at the GRADE Guideline Development Workshop,” said Dunnack. “From the didactic material to the small group sessions, there was so much to learn. The ability to network with other professionals and hear their viewpoints added immensely to the experience."

Dunnack and Lane pose with Dr. Shahnaz Sultan of the U.S. GRADE Network

Giulia Lane: Shared Decision-Making in Urology Guidelines

Dr. Giulia Lane is a urology fellow at the University of Michigan. Lane’s project examines the current uptake and opportunities for improvement in the use of shared decision-making aids in the field of urology. Although shared decision-making aids are a critical step of assessing patients’ values and preferences, Lane explained, they are not commonly used in the field of urology. While almost everyone completes the initial step of informing patients about treatment options, exploring individual preferences is less frequent, and of the 29 American Urological Association (AUA) guidelines reviewed by Dr. Lane, only 12 mention shared decision-making at all. Of those guidelines that mentioned shared decision-making by name, only seven (58%) further described the process. Clinicians in urology, Dr. Lane concluded, are not provided support or encouragement to use shared decision-making in their practice. However, this appears to be changing: shared decision-making was mentioned in 2019 guidelines nearly eight times as often as those published in 2010.

In contrast to what is commonly believed, shared decision-making does not take more time once clinicians are trained. Clinicians should consider providing ready-made online modules to patients to review ahead of their visit, further reducing the time required to improve patient engagement and understanding of their treatment options. Dr. Lane’s work provides a baseline assessment of the use of shared decision-making in urology and will help inform the direction of future research in this area.

“I felt that the best part of the GRADE workshop was the small-group setting with intimate access to experts in the field,” said Dr. Lane. “The networking opportunities were exceptional and I met several people with whom I plan to collaborate. The people I met at the GRADE workshop and the transfer of ideas were the most important take away point for me.”

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Chicago, Illinois will be July 31, 2020.

Evidence Foundation Scholar Update

Contributed by Janice Tufte, 2019 Evidence Foundation Scholar & Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

Janice Tufte, an independent consultant and a leader in patient-public partnership initiatives, received a scholarship to attend the tenth GRADE guideline development workshop held in Denver, Colorado in February 2019 (blog post here). As part of her scholarship, Tufte presented to the larger workshop group on the unique opportunities of using patient partners during the development of GRADE guidelines.

Spring 2019 scholarship recipients: Dr. Irbaz bin Riaz (L) and Ms. Janice Tufte (R)

Tufte has worked with the American College of Physicians as a public panel member on multiple guidelines in addition to serving as a panel member of their Clinical Guidelines Committee. In these roles, she has provided input on outcomes of importance and on future topics for guideline development and is listed as a co-author on recent guidelines and guidance statements. She has also presented to groups on the basics of guideline development including fellow ambassadors of the Patient-Centers Outcomes Research Institute (PCORI).

Recently, Tufte co-published a protocol for the development of guidance for multi-stakeholder engagement (MuSE) in health and healthcare guideline development and implementation. The forthcoming guidance will be based on four different systematic reviews, including an examination of the current barriers and opportunities for stakeholder involvement in the guideline development process and the effect of stakeholder engagement on resulting guidelines and their implementation. The project will ultimately provide recommendations for ways to improve stakeholder engagement.

Here, Tufte provides an update on her work to continue to promote patient involvement in guideline development, including the use of GRADEpro software for these purposes.

My first exposure to the GRADE process and subsequent utilization of the GRADEpro platform genuinely captured and kept my attention. I quickly realized how beneficial both the Evidence to Decision and Summary Tables could be to public patients involved with systematic reviews, guideline development, and recommendations. I could view the overall landscape and better comprehend the pertinent information and findings in the EtD and Summary table that I needed as a non-scientist, allowing me to contribute more effectively in a meaningful manner in a panel conversation on evidence, recommendations or judgements.

GRADEpro is a versatile and modifiable online format. We are able to create a foundation based on the specificities and needs unique to our individual question that will then display the evidence and quality syntheses in a neat table as well as a reliable summary. Having diverse stakeholders at the table is important. Encouraging all to bring in the public and patient perspective is my particular favorite niche, and I will continue to promote this no matter the topic area.

I have recently enjoyed working with the new GRADEpro extensions while serving on a guidelines panel, where we have incorporated GRADE infographics. One challenge I have discovered is synthesizing the information down to an easy understandable one-pager for both clinicians and public reading.

I am looking forward to working with GRADE in the future, with a goal to better deliver reliable, understandable evidence to all who could benefit.

Stay tuned for future updates regarding Janice’s continued work in promoting patient engagement in guideline development.

If you are interested in learning more about GRADE and attending the workshop as a scholarship recipient, applications for our upcoming workshop in Chicago this October are open. The deadline to apply is July 31, 2020. Details can be found here.

Research Shorts: A Theoretical Framework and Competency-Based Approach to Training in Guideline Development

Contributed by Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

As an increasing number of organizations are developing clinical guidelines, expectations for the quality and trustworthiness of these guidelines are on the rise as well. Thus, there is an increased need for guideline-producing organizations to identify, train, and hire or contract with individuals who are adequately skilled in guideline development methods, and for a clear delineation of the knowledge and skill sets required of these individuals.

Recently, seven members of the U.S. GRADE Network and GRADE Working Group co-published an article establishing a framework of core competencies for individuals serving on guideline development panels in a variety of roles, from panel content experts to methodologists. The paper outlines the minimal knowledge, skills, and expertise that would allow an individual to perform tasks adequately. The framework describes three major domains of competency for guideline development: facilitating the development of guideline structure and setup, making judgments about the quality or certainty of evidence, and transforming evidence into recommendations.

Within each core competency, there are multiple sub-competencies and various educational “milestones” which further clarify the required skill. These milestones track to the five stages of educational development established by the Dreyfus model: novice, advanced beginner, competent, proficient, and expert. The authors note that the required level of expertise related to these milestones will vary depending on the role of the individual in developing the guideline. While some sub-competencies related to rating of the certainty of evidence follow the GRADE approach specifically, much of the competency-based framework can be applied universally to other guideline development methodologies. The authors encourage future research efforts to validate, assess, and refine the proposed milestones for their widespread use in efforts to train the next generation of guideline developers.

Sultan S, Morgan RL, Murad MH, Falck-Ytter Y, Dahm P, Schünemann HJ, Mustafa RA. A Theoretical Framework and Competency-Based Approach to Training in Guideline Development. Journal of general internal medicine. 2019 Nov 14:1-7.

Manuscript available here on publisher's site.

Fall 2019 - Scholarship recipients

Contributed by Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

The Eleventh GRADE Guideline Development Workshop was held in Orlando, Florida, this past September. This workshop welcomed 52 participants, including several international attendees from Canada and Brazil. Among these participants were three recipients of a scholarship provided by the Evidence Foundation, which covers the cost of registration. Coming from diverse backgrounds ranging from organizational work to evidence synthesis to policymaking, scholars Faduma Gure, Eric Linskens, and Christian Kershaw presented their proposals for new innovations or opportunities for improving the application and implementation of evidence-based medicine.

Faduma Gure, MSc, a knowledge translation and research specialist for the Association of Ontario Midwives, discussed the challenges of incorporating client perspectives to midwifery guidelines for the organization, which utilizes the GRADE approach. Pregnancy through the post-partum period is often a tumultuous time filled with decision-making, Gure explained, and more can be done to better understand the values and preferences of midwifery clients and to employ an equity lens when formulating recommendations. Gure proposed a solution that includes the development of an equity advisory group consisting of key stakeholders representative of Ontario’s population of midwifery clients. The organization could then involve these stakeholders through the entire guideline development process - from the initial setting of research priorities to the ultimate formulation of recommendations - and elicit important feedback about patient values and preferences as well as the potential impacts of a guideline on various communities.

Eric Linskens, BSc, serves the Minneapolis Veterans Affairs Evidence Synthesis Program and the Minnesota Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center. Linskens presented his current work applying the GRADE approach to existing systematic reviews which did not originally use GRADE. Linskens discussed some of the challenges that his team has faced as part of the initiative, as well as innovative solutions to these issues. For instance, a systematic review conducted by AHRQ may automatically rate down for inconsistency due simply to the inclusion of a sole study, whereas in the GRADE approach, this would not be the case. Additionally, an existing review may break down one clinical question into multiple smaller analyses of comparators or sub-populations, whereas it would be more clinically relevant to use these data to create one larger recommendation. To best solve these issues, Linskens noted, it is important to consider the end-user of any given review or guideline so that their needs can be best met. Additionally, transparently reporting all judgments around the analyses is key. Regarding his time at the workshop, Linskens said, “[i]t was very helpful to work through examples with the GRADE workshop facilitators in small group sessions. They answered our questions as they came up.”

Christian Kershaw, PhD, is a molecular neuroscientist who now works as a health policy analyst for CGS, a Medicare fee-for-service contractor. Dr. Kershaw used her personal experience transitioning from bench science to policymaking to inspire her presentation on the utility of cross-functional teams in medicine and healthcare policy. To develop a cross-functional team, Dr. Kershaw explained, it is best to identify a problem that would best be solved by a group of individuals with heterogeneous skills and backgrounds that would each uniquely serve a common goal or purpose. As an example, Kershaw discussed the development of a team to standardize the way information is used to form coverage decisions as part of the 21st Century Cures Act. The team is comprised of a medical doctor to understand the need for and content of the policies; an outreach and education specialist to understand their legal implications; and a basic research scientist to compile and assess the information. Leveraging individual team members’ strengths and encouraging innovation are keys to success when working in a cross-functional team. “I was impressed with the versatility of the GRADE framework,” Kershaw noted. “It was very informative to learn all of the different ways that the conference attendees were using GRADE to suit their projects.”

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Phoenix, AZ will be December 4, 2019.

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

Contributed by Madelin Siedler, 2018/2019 U.S. GRADE Network Research Fellow

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

One of the benefits of the GRADE approach is that it provides a framework for the development of evidence-based recommendations that are clear and actionable for practicing clinicians even when only lower-quality evidence is available. However, in some particular instances, caution is warranted when developing a recommendation based on low-quality evidence or inference. “Good practice statements” are one such instance. 

The term “good practice statement” is sometimes interchanged for “motherhood statement”.” In either case, the practice being recommended is usually something that is already commonly accepted as beneficial or practical advice. It could even be seen as irrefutably “good” as motherhood and apple pie (hence the term). The nature of these types of statements is such that the action is seen as so obviously beneficial that it would be unduly onerous to conduct a review to demonstrate its efficacy.

An example of a good practice statement is the first recommendation from the American Gastroenterological Association (AGA)’s 2015 guideline on the management of asymptomatic pancreatic cysts, which reads, “The AGA recommends that before starting any pancreatic cyst surveillance program, patients should have a clear understanding of programmatic risks and benefits” (Vege et al., 2015).

How to Spot a Good Practice Statement
An easy way to identify a good practice statement is to restate the recommendation as its inverse: for instance, “patients should not have a clear understanding of programmatic risks and benefits.” If this “unstated alternative” is absurd or clearly does not conform to ethical norms, the original statement is likely a good practice statement (Guyatt et al., 2016).

The Problem with Good Practice Statements
The GRADE Working Group recommends that good practice statements be used sparingly, if at all. Because good practice guidance is typically based on several linked sources of indirect evidence, there is no way to tell whether the benefits of the proposed recommended action are as truly obvious or incontestable as they seem. And if they are (if the inverse of the proposed recommendation would be absurd or unethical) then they are likely unwarranted and can dilute the strength of the guideline as a whole. 

Sometimes, good practice statements may even appear as graded recommendations in a guideline (a decision that’s not recommended by the GRADE working group for the reasons above). In this case, the guideline authors may be tempted to make a strong recommendation based on low-quality evidence, which should ideally be a rare occurrence and based on well-defined criteria (Guyatt et al., 2015). 

Practical Advice for Dealing with Good Practice Statements
The GRADE Working Group recommends using the following checklist to determine whether a good practice statement is warranted:

1. Is the statement clear and actionable? 
2. Is the message really necessary in regards to actual health practice? 
3. After consideration of all relevant health outcomes and potential downstream consequences, will implementing the good practice statement result in large net positive consequences? 
4. Is collecting and summarizing the evidence a poor use of a guideline panel’s limited time and energy? 
5. Is there a well-documented clear and explicit rationale connecting the indirect evidence?

If the answer is 'yes' to all five questions, a good practice statement may be warranted for inclusion in a guideline document. When done correctly, good practice advice statements should appear as ungraded recommendations, meaning no formal rating of quality of evidence or strength of recommendation should be given (Guyatt et al., 2015).

However, many potential good practice statements will be eliminated through the use of this checklist. For instance, careful consideration of Question #3 could lead the panel to realize that the assumed net positive of a specific action may not be so obvious after all. In this case, the guideline panel should consider whether a thorough review of the evidence should be conducted and formal grading methods applied.

Fall 2018 Scholarship Recipients

Contributed by Madelin Siedler, 2018/2019 U.S GRADE Network Research Fellow

We were pleased to support the participation of three research scholars at our recent GRADE guideline development workshop held in Silver Spring, Maryland, October 17-19, 2018. By providing complementary registration, we hope that these scholars increased their understanding and application of the GRADE approach. As part of the U.S. GRADE Network/Evidence Foundation scholarship, recipients presented to fellow workshop participants about their interests and current endeavors in the field of guideline development.

Scholarship recipients: Oilvia Magwood, Mohamad Kalot, and Mohammed Alkhatib

A few details about our Fall 2018 scholarship recipients:

Mohamad Kalot, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in improving quality and decreasing disparities in healthcare through the development of evidence-based guidelines. Dr. Kalot presented on his current work in the development of guidelines for the management of rare diseases. Dr. Kalot explained how the development of such guidelines presents a unique challenge in that there is often a dearth of research on these populations, which can affect the directness of evidence among other factors.

“Since an important part of my conducted reviews and research deals with developing guidelines for rare diseases, I would ultimately face specific challenges in the process of assessing the certainty of evidence in my work, and I’m interested in exploring innovative methods to deal with these challenges,” said Kalot. “The GRADE workshop in Silver Spring didn’t only help me find solutions for my challenges, it made me see the research methodology world from a different, deeper and more practical perspective - especially with the very helpful tools [RevMan and GRADEpro Guideline Development Tool] that we learned about in the small groups sessions and the discussions about rating up and rating down the quality of evidence.”

Olivia Magwood, MPH, attended from the Buyere Research Institute in Ottawa, Ontario, where she has participated in the development of three national and international guidelines. Ms. Magwood presented on the development of the FACE (Feasibility, Acceptability, Cost, and Equity) Framework Stakeholder Survey. The FACE Stakeholder Survey aims to identify cognitive bias that may influence guideline development as well as inform the quality and impact and improve the uptake of evidence-based recommendations in various stakeholder groups.

“For me, the GRADE workshop highlighted the importance of building my network and having the right people on your team throughout the guideline development process,” explained Magwood. “This includes involving methodologists early and throughout guideline development, as well as emphasizing patient perspectives, especially while deciding on patient-important outcomes. Guideline development is truly a collaborative process that benefits from multi-stakeholder engagement.”

Mohammed Alkhatib, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in examining and refining the adoption, adaptation and de novo development (or “adolopment,” for short) of guidelines to be used in lower-resource settings. Dr. Alkhatib presented on the importance of this approach, as well as the unique challenges of tailoring the development of clinical recommendations to settings with limited resources, such as developing nations and areas of conflict.

“It was an exceptional experience for me to present my proposal in front of high-level scholars of guideline development and to listen to their positive feedback,” said Alkhatib. “[The] GRADE workshop was very helpful for me at the level of interpretation of [systematic reviews] and how to judge strengths and weaknesses using GRADEpro.”

**If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Denver, Colorado will be January 1, 2019.

Research Shorts: Surrogate endpoints using the example of hepatitis C virus

Contributed by Claudia Dobler, MD, PhD

2018 U.S. GRADE Workshop Scholarship Recipient

Surrogate endpoints (for example laboratory or imaging results) are commonly used in clinical trials, as they require less participants and can be done in a shorter period of time compared to trials that use clinically important outcome measures such as mortality. When evidence for an intervention is almost exclusively based on trials that used surrogate outcomes, it is challenging for decision makers to determine the appropriateness of the use and reimbursement of an intervention. The common tendency in evidence-based medicine is to view results based on surrogate endpoints as less certain than results based on long term, final patient-important outcomes. The authors of this paper use the contemporary and highly debated example of the surrogate endpoint ‘sustained viral response’ (i.e., viral eradication considered to represent successful treatment) in patients treated for chronic hepatitis C virus infection to demonstrate how the validity of a surrogate endpoint can be critically appraised to assess the trustworthiness of the evidence and the implications for decision-making. They outline how the GRADE system for determining the certainty in the evidence can be used in situations where decisions for clinical practice and health policy have to be based on evidence that mainly comes from trials with indirect outcome measures.

Beyond assessing the quality of the evidence, potential benefits and harms of the intervention need to be weighed against each other and factors such as patient values, impact on healthcare equity, acceptability by patients and feasibility of the intervention need to be considered. The authors conclude that considering all these factors, a conditional recommendation for direct acting antiviral agents to treat chronic hepatitis C virus infection may be appropriate.


Reference: Dobler CC, Morgan RL, Falck-Ytter Y, Montori VM, Murad MH. Assessing the validity of surrogate endpoints in the context of a controversy about the measurement of effectiveness of hepatitis C virus treatment. BMJ evidence-based medicine 2018: 23(2): 50-53. https://ebm.bmj.com/content/22/6/199

Research Shorts: Treatment Burden

Contribution from Claudia Dobler, MD, PhD

2018 U.S. GRADE Workshop Scholarship Recipient

Treatment burden is the work that patients need to do to implement treatments and its effect on patient functioning and well-being. The work that patients do to manage their health often remains invisible to health professionals, who might therefore overestimate patient’s capacity to take on more interventions. Treatment burden includes filling prescriptions, taking medications, attending medical appointments, monitoring their health, making lifestyle or behavioral changes, managing medical equipment or devices, and having to deal with insurance companies and the financial burden of health care. The evidence to decision framework developed by the GRADE Working Group addresses some parts of the construct of burden of treatment in the domains of acceptability and feasibility, but burden of treatment is currently not included as a discrete construct. The authors of this paper explore the idea of including information on treatment burden associated with interventions in clinical practice guidelines. Information on treatment burden would enable patients to make informed decisions about treatments. High-quality methods for assessing treatment burden need to be identified and meaningful ways of adding this information to clinical practice guidelines need to be explored. In the paper, practical examples are outlined of how treatment burden could be included in guidelines for chronic obstructive pulmonary disease, type 2 diabetes, and chronic kidney disease. Making treatment burden explicit for adhering to a special diet for chronic kidney disease, for example, could include information on foods that should be avoided, extra time spent on grocery shopping and cooking, challenges when eating out, the typical number of consultations with a dietitian, transport requirements to see a dietitian, time spent documenting the diet, and required intensity of blood tests to monitor electrolytes.




Clinicians should discuss values and preferences with patients, especially for treatments associated with a high workload. This would help clinicians to understand how individual patients prioritize different outcomes and treatment burden and would help patients to understand what components of their treatment are the essentials, even on a bad day, so that they can prioritize treatments appropriately.


Dobler CC, Harb N, Maguire CA, Armour CL, Coleman C, Murad MH. Treatment burden should be included in clinical practice guidelines. BMJ 2018: 363. https://www.bmj.com/content/363/bmj.k4065

Drug Value Symposium

The first Drug Value Symposium was held in Cleveland, Ohio on May 5th, 2018 in conjunction with the 8th Guideline Development Workshop using the GRADE approach.

Attendees found the inaugural Drug Value Symposium to be engaging, appreciating the opportunity to share ideas on how to systematically approach making evidence-based drug-coverage decisions at their institutions.

Attendees from all over the world with various backgrounds and expertise attended. The symposium opened with a thought-provoking presentation from Yngve Falck-Ytter, one of the founding U.S. GRADE Network members. His presentation highlighted why healthcare costs are becoming an increasing challenge and how incorporating a methodologically rigorous framework, such as GRADE, to formulary decision making can facilitate sensible decisions. 

Dr. Falck-Ytter opens the symposium with an overview of how GRADE includes both cost and cost-effectiveness when making health-care decisions.

Representatives from local healthcare systems then offered varying perspectives on the institutional successes and challenges of managing high cost drugs. Presenters included Mandy Leonard from Cleveland Clinic, Indrani Kar from University Hospitals, Christina Wadsworth and Jan Kover from MetroHealth, and Candice Wenzell from the Northeast Ohio VA Healthcare System. 

       Lessons learned from managing costly drugs within the Cleveland, OH 

Veteran's Administration, as presented by Ms. Wenzell.

The symposium concluded with an interactive session facilitated by Yngve Falck-Ytter, Candice Wenzell, and Kelsey Rife that allowed attendees to apply the GRADE methodology to evaluate a drug for a specific indication and then make a decision on whether or not they would recommend the use of the drug. 

Join us in Cleveland, OH!

Our next Guideline Development Workshop using the GRADE approach will be held in Cleveland, OH from May 2nd through 4th, 2018. More information, including a proposed agenda, registration, and hotel information can be found at http://gradeconf.org/. As with every workshop, we encourage you to bring examples of your work and questions to discuss with participants at the workshop and the faculty.



We hope to see you there!

It's been a DTA kind of year

We started 2016 in Minneapolis by hosting our regular Guideline Development Workshop focused on making recommendations about diagnostic tests and test strategies. In 2016, the GRADE Working Group released guidance on decision making about diagnostic tests. That paper can be found here. Our Spring workshop featured three days of guideline development materials tailored for tests and test strategies, including a pre-course systematic review workshop on how to meta-analyzing this material. Dr. Holger Schünemann provided guest lectures and support during the small group exercises.



Earlier this week, in collaboration with the Dutch GRADE Network and Barcelona GRADE Center, Dr. Reem Mustafa kicked off a similar workshop on diagnostic test accuracy (DTA).



Many resources have been released detailing the advancements in DTA decision making. A few are listed below:

• Decision-making about healthcare-related tests and diagnostic strategies series. Paper 1: testing to improve people's health: a new series
• Decision-Making about Healthcare Related Tests and Diagnostic Strategies: User Testing of GRADE Evidence Tables

Proposed new evidence-based medicine pyramid

From: MH Murad, N Asi, M Alsawas, F Alahdab. Perspective: New evidence pyramid. Evid Based Med. Online First: June 20, 2016. 

Systematic reviews and meta-analyses have been placed at the top of the evidence pyramid for several good reasons. They provide more trustworthy answers and more precise estimates with narrower confidence intervals that are were not selected based on expert opinion, but rather based on a systematic procedure. However, credible systematic review can summarize biased evidence and poorly done systematic reviews can summarize well done trials. This challenges the placement on top. In addition, GRADE tells us that our certainty in evidence should be driven by many factors other than study design. Therefore, we propose 2 modifications to the pyramid to resolve these 2 challenges.

Figure 1.

Legend: The proposed new evidence-based medicine pyramid. (A) The traditional pyramid. (B) Revising the pyramid: (1) lines separating the study designs become wavy (Grading of Recommendations Assessment, Development and Evaluation), (2) systematic reviews are ‘chopped off’ the pyramid. (C) The revised pyramid: systematic reviews are a lens through which evidence is viewed (and applied).

Announcing our fall Guideline Development Workshop in Orlando Nov 2-4, 2016

As our workshop last month in Kansas City, Missouri, sold out 4 weeks before the event, we have now opened registration to the fall workshop early to facilitate better planning! Please visit our conference website for details.

New this year we is a half day pre-workshop activity to learn more about the role of Systematic Reviews: increase your comfort level with Systematic Reviews and even learn the basics of RevMan, the Cochrane Collaboration's software for meta-analysis. This pre-workshop is specifically designed to dovetail with the GRADE guidelines workshop and we have introduced this option based on popular demand.

Annoucement: Spring 2016 - Guideline Development Workshop

Calling all interested in learning about guideline development using the GRADE approach...
Our next Guideline Development Workshop featuring the GRADE Approach will be held April 20-22, 2016 in Kansas City, Missouri! Details about the workshop, including the schedule, tutors, and a link to register can be found at GRADEconf.org. Kansas City, Missouri (not to be confused with their neighbor Kansas City, Kansas), is known for having 200 fountains in the metropolitan area, only second to the number of fountains in Rome. In addition to beautiful architecture, Kansas City is now home to the 2015 World Series Champions! Go Royals! We look forward to seeing you in the great city of KC in April!