GRADE Guideline Development Workshop - now with INGUIDE!

We at the U.S. GRADE Network are excited to announce that, for the first time ever, we have partnered with INGUIDE, a credentialing system for guideline developers, to offer free access to their Level 1 online training to participants of our Guideline Development Workshop and Guideline Fundamentals for Panel Members and Organization Staff pre-workshop course. 

This announcement comes just in time for our next workshop, held in Austin, Texas May 17-19, for which there are still spots available at our workshop website.

The INGUIDE program, the result of a collaboration between the Guidelines International Network (G-I-N) and McMaster University, provides certification of individuals involved in guideline development on four successive levels, from a Guideline Panel Member to becoming a Guideline Development Credentialing Instructor themselves. 

To obtain access to the INGUIDE Level 1 course, simply register for and attend our 3-day GRADE guideline development workshop in addition to the pre-workshop course on Guideline Fundamentals for Panel Members and Organization Staff. This 3-hour pre-workshop course will cover all of the necessary information about the practical aspects of guideline development, including:

• the importance of trustworthy guidelines
• checklists for the development of guidelines
• the structure and roles of a guideline panel
• the process of grading evidence and formulating recommendations

The main 3-day workshop will cover important aspects of assessing risk of bias, rating the certainty of evidence, and considering other important factors when using GRADE to develop a guideline.

Our pre-workshop course combined with our main three-day in-person course will provide all of the training required for a breeze-through experience online. You will receive free access to the ~2-3 hour online course, comprising four modules, that you can complete at your convenience after the workshop. Upon completion, you will be officially certified as an INGUIDE Level 1 Guideline Panel Member.

Why We Can't Wait for Our Austin Workshop

The U.S. GRADE Network's 18th workshop is just five weeks away! With a focus on assessing certainty of evidence informed by non-randomized studies, this workshop will take place in Austin, Texas - the home of Austin City Limits, the famous Bat Bridge, and some next-level barbecue joints.

We asked some of the workshop facilitators about what they're looking forward to about the trip (besides the chance to talk about GRADE and evidence-based medicine for three days straight!) Here's what they said...

Limited space remains for the workshop. Learn more and register and www.gradeconf.org. 

2022 Recap and What's Next for the U.S. GRADE Network

Thanks to you, 2022 was one of our best years yet.

We were relieved to see you in person, our first time since 2020, at our spring workshop in Chicago last June! Then, we reconvened online for a virtual workshop in November. Overall, we hosted over 100 participants from around the world, stretching from Canada and Spain to Colombia and Sweden. 

Facilitators and participants at the spring 2022 GRADE guideline development workshop in Chicago, IL - the first in-person USGN workshop since March 2020.

We also launched our new two-day virtual Systematic Review workshop offering and welcomed another 48 learners, including a record 13 scholars from across the globe!

We published our takeaways from our early experiences moving our workshop online in response to the Covid-19 pandemic in BMJ Evidence-Based Medicine, which you can read about here.

Finally, we launched the USGN Webinar Series and will host regular free webinars on all things GRADE. For our inaugural session, we were joined by Drs. Gordon Guyatt and Reem Mustafa. You can watch the recording and subscribe to our new YouTube channel here. 

We look forward to seeing you at our 18th Guideline Development Workshop, in person in sunny Austin, Texas, May 17-19, 2023! The workshop will have a special focus on assessing the certainty of evidence informed by non-randomized studies. Registration is now open.

U.S. GRADE Network Holds Its First Two-Day Systematic Review Workshop

In response to popular demand, the U.S. GRADE Network recently expanded its half-day course on systematic reviews into a two-day virtual workshop. Taking place over February 1-2, 2022, the sessions comprised six total hours of instructional time and question-and-answer sessions with Network faculty in addition to three hours of hands-on activities in a small-group format. Large-group lectures ranged from developing a focused clinical question to conducting meta-analysis and evaluating the quality of systematic reviews. In hands-on sessions of ten participants each, attendees were introduced to a free online screening platform (Rayyan) and tried their hands at assessing risk of bias and conducting meta-analysis in RevMan, a free systematic review and meta-analysis software from Cochrane.

As part of the program, the Evidence Foundation welcomed three recipients of scholarships to attend the workshop free of charge. As part of their applications, the scholars described a current or proposed project for a systematic review. The three recipients, along with their projects, included:

• Bryden Giving, MAOT, OTR/L (Boston University): A traffic light of evidence for occupational therapy interventions supporting autistic children and youth
• Nirjhar Ruth Ghosh, MS (Texas A&M University): Evidence-based practice in the field of nutrition: A systematic review of knowledge, skills, attitudes, behaviors and teaching strategies
• Milton A. Romero-Robles (Universidad Nacional del Santa): Participation, involvement and main barriers in the inclusion of patients with non-communicable diseases in the development of clinical practice guidelines: A systematic review protocol

Be the first to hear about these and other trainings @USGRADEnet on twitter or at www.systematicreview.org.

Note: applications for scholarships to attend the upcoming GRADE Guideline Development Workshop held July 29-July, 2022, in Chicago, Illinois, close March 31. See application details here.

Spring 2020 - Scholarship Recipients

Earlier this month, the U.S. GRADE Network welcomed 51 participants to the Twelfth GRADE Guideline Workshop in sunny Phoenix, Arizona. Two participants, Giulia Lane and Hayley Dunnack, were awarded a scholarship from the Evidence Foundation that covered their workshop registration fee. As the two latest additions in a line of over 40 Foundation scholars since 2014, Lane and Dunnack presented to their fellow participants about their projects related to improving the use of evidence-based decision-making in healthcare.

Hayley Dunnack, left, and Giulia Lane, right, were the recipients of the Evidence Foundation scholarship for the spring 2020 GRADE workshop held in Phoenix, AZ

Hayley Dunnack: Opportunities and Challenges in Guidelines for Nursing Practice

Hayley Dunnack, BSN, CMS-RN, OCN is a PhD candidate at the University of Connecticut School of Nursing. Dunnack presented on her work with the Oncology Nursing Society (ONS) to develop guidelines on symptom management for patients with cancer. Currently, ONS has several projects underway to develop guidelines on symptom management based on systematic reviews of the evidence and using GRADE guideline methodology. A current challenge in the ONS’ guideline development relates to the development of guidelines on topics for which randomized studies are sparse or difficult to conduct, such as the management of hypersensitivity reactions or the use of vascular access devices.  Dunnack’s project aims to examine the use of non-randomized and “real-world” evidence to inform the development of rigorous guidelines based on systematic reviews and using the GRADE process.

“I had an incredible experience at the GRADE Guideline Development Workshop,” said Dunnack. “From the didactic material to the small group sessions, there was so much to learn. The ability to network with other professionals and hear their viewpoints added immensely to the experience."

Dunnack and Lane pose with Dr. Shahnaz Sultan of the U.S. GRADE Network

Giulia Lane: Shared Decision-Making in Urology Guidelines

Dr. Giulia Lane is a urology fellow at the University of Michigan. Lane’s project examines the current uptake and opportunities for improvement in the use of shared decision-making aids in the field of urology. Although shared decision-making aids are a critical step of assessing patients’ values and preferences, Lane explained, they are not commonly used in the field of urology. While almost everyone completes the initial step of informing patients about treatment options, exploring individual preferences is less frequent, and of the 29 American Urological Association (AUA) guidelines reviewed by Dr. Lane, only 12 mention shared decision-making at all. Of those guidelines that mentioned shared decision-making by name, only seven (58%) further described the process. Clinicians in urology, Dr. Lane concluded, are not provided support or encouragement to use shared decision-making in their practice. However, this appears to be changing: shared decision-making was mentioned in 2019 guidelines nearly eight times as often as those published in 2010.

In contrast to what is commonly believed, shared decision-making does not take more time once clinicians are trained. Clinicians should consider providing ready-made online modules to patients to review ahead of their visit, further reducing the time required to improve patient engagement and understanding of their treatment options. Dr. Lane’s work provides a baseline assessment of the use of shared decision-making in urology and will help inform the direction of future research in this area.

“I felt that the best part of the GRADE workshop was the small-group setting with intimate access to experts in the field,” said Dr. Lane. “The networking opportunities were exceptional and I met several people with whom I plan to collaborate. The people I met at the GRADE workshop and the transfer of ideas were the most important take away point for me.”

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Chicago, Illinois will be July 31, 2020.

Research Shorts: A Theoretical Framework and Competency-Based Approach to Training in Guideline Development

Contributed by Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

As an increasing number of organizations are developing clinical guidelines, expectations for the quality and trustworthiness of these guidelines are on the rise as well. Thus, there is an increased need for guideline-producing organizations to identify, train, and hire or contract with individuals who are adequately skilled in guideline development methods, and for a clear delineation of the knowledge and skill sets required of these individuals.

Recently, seven members of the U.S. GRADE Network and GRADE Working Group co-published an article establishing a framework of core competencies for individuals serving on guideline development panels in a variety of roles, from panel content experts to methodologists. The paper outlines the minimal knowledge, skills, and expertise that would allow an individual to perform tasks adequately. The framework describes three major domains of competency for guideline development: facilitating the development of guideline structure and setup, making judgments about the quality or certainty of evidence, and transforming evidence into recommendations.

Within each core competency, there are multiple sub-competencies and various educational “milestones” which further clarify the required skill. These milestones track to the five stages of educational development established by the Dreyfus model: novice, advanced beginner, competent, proficient, and expert. The authors note that the required level of expertise related to these milestones will vary depending on the role of the individual in developing the guideline. While some sub-competencies related to rating of the certainty of evidence follow the GRADE approach specifically, much of the competency-based framework can be applied universally to other guideline development methodologies. The authors encourage future research efforts to validate, assess, and refine the proposed milestones for their widespread use in efforts to train the next generation of guideline developers.

Sultan S, Morgan RL, Murad MH, Falck-Ytter Y, Dahm P, Schünemann HJ, Mustafa RA. A Theoretical Framework and Competency-Based Approach to Training in Guideline Development. Journal of general internal medicine. 2019 Nov 14:1-7.

Manuscript available here on publisher's site.

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

Contributed by Madelin Siedler, 2018/2019 U.S. GRADE Network Research Fellow

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

One of the benefits of the GRADE approach is that it provides a framework for the development of evidence-based recommendations that are clear and actionable for practicing clinicians even when only lower-quality evidence is available. However, in some particular instances, caution is warranted when developing a recommendation based on low-quality evidence or inference. “Good practice statements” are one such instance. 

The term “good practice statement” is sometimes interchanged for “motherhood statement”.” In either case, the practice being recommended is usually something that is already commonly accepted as beneficial or practical advice. It could even be seen as irrefutably “good” as motherhood and apple pie (hence the term). The nature of these types of statements is such that the action is seen as so obviously beneficial that it would be unduly onerous to conduct a review to demonstrate its efficacy.

An example of a good practice statement is the first recommendation from the American Gastroenterological Association (AGA)’s 2015 guideline on the management of asymptomatic pancreatic cysts, which reads, “The AGA recommends that before starting any pancreatic cyst surveillance program, patients should have a clear understanding of programmatic risks and benefits” (Vege et al., 2015).

How to Spot a Good Practice Statement
An easy way to identify a good practice statement is to restate the recommendation as its inverse: for instance, “patients should not have a clear understanding of programmatic risks and benefits.” If this “unstated alternative” is absurd or clearly does not conform to ethical norms, the original statement is likely a good practice statement (Guyatt et al., 2016).

The Problem with Good Practice Statements
The GRADE Working Group recommends that good practice statements be used sparingly, if at all. Because good practice guidance is typically based on several linked sources of indirect evidence, there is no way to tell whether the benefits of the proposed recommended action are as truly obvious or incontestable as they seem. And if they are (if the inverse of the proposed recommendation would be absurd or unethical) then they are likely unwarranted and can dilute the strength of the guideline as a whole. 

Sometimes, good practice statements may even appear as graded recommendations in a guideline (a decision that’s not recommended by the GRADE working group for the reasons above). In this case, the guideline authors may be tempted to make a strong recommendation based on low-quality evidence, which should ideally be a rare occurrence and based on well-defined criteria (Guyatt et al., 2015). 

Practical Advice for Dealing with Good Practice Statements
The GRADE Working Group recommends using the following checklist to determine whether a good practice statement is warranted:

1. Is the statement clear and actionable? 
2. Is the message really necessary in regards to actual health practice? 
3. After consideration of all relevant health outcomes and potential downstream consequences, will implementing the good practice statement result in large net positive consequences? 
4. Is collecting and summarizing the evidence a poor use of a guideline panel’s limited time and energy? 
5. Is there a well-documented clear and explicit rationale connecting the indirect evidence?

If the answer is 'yes' to all five questions, a good practice statement may be warranted for inclusion in a guideline document. When done correctly, good practice advice statements should appear as ungraded recommendations, meaning no formal rating of quality of evidence or strength of recommendation should be given (Guyatt et al., 2015).

However, many potential good practice statements will be eliminated through the use of this checklist. For instance, careful consideration of Question #3 could lead the panel to realize that the assumed net positive of a specific action may not be so obvious after all. In this case, the guideline panel should consider whether a thorough review of the evidence should be conducted and formal grading methods applied.

Fall 2018 Scholarship Recipients

Contributed by Madelin Siedler, 2018/2019 U.S GRADE Network Research Fellow

We were pleased to support the participation of three research scholars at our recent GRADE guideline development workshop held in Silver Spring, Maryland, October 17-19, 2018. By providing complementary registration, we hope that these scholars increased their understanding and application of the GRADE approach. As part of the U.S. GRADE Network/Evidence Foundation scholarship, recipients presented to fellow workshop participants about their interests and current endeavors in the field of guideline development.

Scholarship recipients: Oilvia Magwood, Mohamad Kalot, and Mohammed Alkhatib

A few details about our Fall 2018 scholarship recipients:

Mohamad Kalot, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in improving quality and decreasing disparities in healthcare through the development of evidence-based guidelines. Dr. Kalot presented on his current work in the development of guidelines for the management of rare diseases. Dr. Kalot explained how the development of such guidelines presents a unique challenge in that there is often a dearth of research on these populations, which can affect the directness of evidence among other factors.

“Since an important part of my conducted reviews and research deals with developing guidelines for rare diseases, I would ultimately face specific challenges in the process of assessing the certainty of evidence in my work, and I’m interested in exploring innovative methods to deal with these challenges,” said Kalot. “The GRADE workshop in Silver Spring didn’t only help me find solutions for my challenges, it made me see the research methodology world from a different, deeper and more practical perspective - especially with the very helpful tools [RevMan and GRADEpro Guideline Development Tool] that we learned about in the small groups sessions and the discussions about rating up and rating down the quality of evidence.”

Olivia Magwood, MPH, attended from the Buyere Research Institute in Ottawa, Ontario, where she has participated in the development of three national and international guidelines. Ms. Magwood presented on the development of the FACE (Feasibility, Acceptability, Cost, and Equity) Framework Stakeholder Survey. The FACE Stakeholder Survey aims to identify cognitive bias that may influence guideline development as well as inform the quality and impact and improve the uptake of evidence-based recommendations in various stakeholder groups.

“For me, the GRADE workshop highlighted the importance of building my network and having the right people on your team throughout the guideline development process,” explained Magwood. “This includes involving methodologists early and throughout guideline development, as well as emphasizing patient perspectives, especially while deciding on patient-important outcomes. Guideline development is truly a collaborative process that benefits from multi-stakeholder engagement.”

Mohammed Alkhatib, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in examining and refining the adoption, adaptation and de novo development (or “adolopment,” for short) of guidelines to be used in lower-resource settings. Dr. Alkhatib presented on the importance of this approach, as well as the unique challenges of tailoring the development of clinical recommendations to settings with limited resources, such as developing nations and areas of conflict.

“It was an exceptional experience for me to present my proposal in front of high-level scholars of guideline development and to listen to their positive feedback,” said Alkhatib. “[The] GRADE workshop was very helpful for me at the level of interpretation of [systematic reviews] and how to judge strengths and weaknesses using GRADEpro.”

**If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Denver, Colorado will be January 1, 2019.

Research Shorts: Surrogate endpoints using the example of hepatitis C virus

Contributed by Claudia Dobler, MD, PhD

2018 U.S. GRADE Workshop Scholarship Recipient

Surrogate endpoints (for example laboratory or imaging results) are commonly used in clinical trials, as they require less participants and can be done in a shorter period of time compared to trials that use clinically important outcome measures such as mortality. When evidence for an intervention is almost exclusively based on trials that used surrogate outcomes, it is challenging for decision makers to determine the appropriateness of the use and reimbursement of an intervention. The common tendency in evidence-based medicine is to view results based on surrogate endpoints as less certain than results based on long term, final patient-important outcomes. The authors of this paper use the contemporary and highly debated example of the surrogate endpoint ‘sustained viral response’ (i.e., viral eradication considered to represent successful treatment) in patients treated for chronic hepatitis C virus infection to demonstrate how the validity of a surrogate endpoint can be critically appraised to assess the trustworthiness of the evidence and the implications for decision-making. They outline how the GRADE system for determining the certainty in the evidence can be used in situations where decisions for clinical practice and health policy have to be based on evidence that mainly comes from trials with indirect outcome measures.

Beyond assessing the quality of the evidence, potential benefits and harms of the intervention need to be weighed against each other and factors such as patient values, impact on healthcare equity, acceptability by patients and feasibility of the intervention need to be considered. The authors conclude that considering all these factors, a conditional recommendation for direct acting antiviral agents to treat chronic hepatitis C virus infection may be appropriate.


Reference: Dobler CC, Morgan RL, Falck-Ytter Y, Montori VM, Murad MH. Assessing the validity of surrogate endpoints in the context of a controversy about the measurement of effectiveness of hepatitis C virus treatment. BMJ evidence-based medicine 2018: 23(2): 50-53. https://ebm.bmj.com/content/22/6/199

Research Shorts: Treatment Burden

Contribution from Claudia Dobler, MD, PhD

2018 U.S. GRADE Workshop Scholarship Recipient

Treatment burden is the work that patients need to do to implement treatments and its effect on patient functioning and well-being. The work that patients do to manage their health often remains invisible to health professionals, who might therefore overestimate patient’s capacity to take on more interventions. Treatment burden includes filling prescriptions, taking medications, attending medical appointments, monitoring their health, making lifestyle or behavioral changes, managing medical equipment or devices, and having to deal with insurance companies and the financial burden of health care. The evidence to decision framework developed by the GRADE Working Group addresses some parts of the construct of burden of treatment in the domains of acceptability and feasibility, but burden of treatment is currently not included as a discrete construct. The authors of this paper explore the idea of including information on treatment burden associated with interventions in clinical practice guidelines. Information on treatment burden would enable patients to make informed decisions about treatments. High-quality methods for assessing treatment burden need to be identified and meaningful ways of adding this information to clinical practice guidelines need to be explored. In the paper, practical examples are outlined of how treatment burden could be included in guidelines for chronic obstructive pulmonary disease, type 2 diabetes, and chronic kidney disease. Making treatment burden explicit for adhering to a special diet for chronic kidney disease, for example, could include information on foods that should be avoided, extra time spent on grocery shopping and cooking, challenges when eating out, the typical number of consultations with a dietitian, transport requirements to see a dietitian, time spent documenting the diet, and required intensity of blood tests to monitor electrolytes.




Clinicians should discuss values and preferences with patients, especially for treatments associated with a high workload. This would help clinicians to understand how individual patients prioritize different outcomes and treatment burden and would help patients to understand what components of their treatment are the essentials, even on a bad day, so that they can prioritize treatments appropriately.


Dobler CC, Harb N, Maguire CA, Armour CL, Coleman C, Murad MH. Treatment burden should be included in clinical practice guidelines. BMJ 2018: 363. https://www.bmj.com/content/363/bmj.k4065

Join us in Cleveland, OH!

Our next Guideline Development Workshop using the GRADE approach will be held in Cleveland, OH from May 2nd through 4th, 2018. More information, including a proposed agenda, registration, and hotel information can be found at http://gradeconf.org/. As with every workshop, we encourage you to bring examples of your work and questions to discuss with participants at the workshop and the faculty.



We hope to see you there!

Proposed new evidence-based medicine pyramid

From: MH Murad, N Asi, M Alsawas, F Alahdab. Perspective: New evidence pyramid. Evid Based Med. Online First: June 20, 2016. 

Systematic reviews and meta-analyses have been placed at the top of the evidence pyramid for several good reasons. They provide more trustworthy answers and more precise estimates with narrower confidence intervals that are were not selected based on expert opinion, but rather based on a systematic procedure. However, credible systematic review can summarize biased evidence and poorly done systematic reviews can summarize well done trials. This challenges the placement on top. In addition, GRADE tells us that our certainty in evidence should be driven by many factors other than study design. Therefore, we propose 2 modifications to the pyramid to resolve these 2 challenges.

Figure 1.

Legend: The proposed new evidence-based medicine pyramid. (A) The traditional pyramid. (B) Revising the pyramid: (1) lines separating the study designs become wavy (Grading of Recommendations Assessment, Development and Evaluation), (2) systematic reviews are ‘chopped off’ the pyramid. (C) The revised pyramid: systematic reviews are a lens through which evidence is viewed (and applied).

Announcing our fall Guideline Development Workshop in Orlando Nov 2-4, 2016

As our workshop last month in Kansas City, Missouri, sold out 4 weeks before the event, we have now opened registration to the fall workshop early to facilitate better planning! Please visit our conference website for details.

New this year we is a half day pre-workshop activity to learn more about the role of Systematic Reviews: increase your comfort level with Systematic Reviews and even learn the basics of RevMan, the Cochrane Collaboration's software for meta-analysis. This pre-workshop is specifically designed to dovetail with the GRADE guidelines workshop and we have introduced this option based on popular demand.

A year in review

A few highlights for the U.S. GRADE Network in 2015:

We hosted our second Guideline Development Workshop in partnership with U.S. Cochrane West (a branch of the U.S. Cochrane Center) in Portland, Oregon. The turnout was so large, it was hard to capture everyone in one photo! 

We hosted our third Guideline Development Workshop in Orlando, FL! Fun fact: The very first USGN Guideline Development Workshop was also held in Orlando in the fall of 2014. We had a great year and look forward to more activities in 2016!