Research Shorts: A Theoretical Framework and Competency-Based Approach to Training in Guideline Development

Contributed by Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

As an increasing number of organizations are developing clinical guidelines, expectations for the quality and trustworthiness of these guidelines are on the rise as well. Thus, there is an increased need for guideline-producing organizations to identify, train, and hire or contract with individuals who are adequately skilled in guideline development methods, and for a clear delineation of the knowledge and skill sets required of these individuals.

Recently, seven members of the U.S. GRADE Network and GRADE Working Group co-published an article establishing a framework of core competencies for individuals serving on guideline development panels in a variety of roles, from panel content experts to methodologists. The paper outlines the minimal knowledge, skills, and expertise that would allow an individual to perform tasks adequately. The framework describes three major domains of competency for guideline development: facilitating the development of guideline structure and setup, making judgments about the quality or certainty of evidence, and transforming evidence into recommendations.

Within each core competency, there are multiple sub-competencies and various educational “milestones” which further clarify the required skill. These milestones track to the five stages of educational development established by the Dreyfus model: novice, advanced beginner, competent, proficient, and expert. The authors note that the required level of expertise related to these milestones will vary depending on the role of the individual in developing the guideline. While some sub-competencies related to rating of the certainty of evidence follow the GRADE approach specifically, much of the competency-based framework can be applied universally to other guideline development methodologies. The authors encourage future research efforts to validate, assess, and refine the proposed milestones for their widespread use in efforts to train the next generation of guideline developers.

Sultan S, Morgan RL, Murad MH, Falck-Ytter Y, Dahm P, Schünemann HJ, Mustafa RA. A Theoretical Framework and Competency-Based Approach to Training in Guideline Development. Journal of general internal medicine. 2019 Nov 14:1-7.

Manuscript available here on publisher's site.

Fall 2019 - Scholarship recipients

Contributed by Madelin Siedler, 2019/2020 U.S. GRADE Network Research Fellow

The Eleventh GRADE Guideline Development Workshop was held in Orlando, Florida, this past September. This workshop welcomed 52 participants, including several international attendees from Canada and Brazil. Among these participants were three recipients of a scholarship provided by the Evidence Foundation, which covers the cost of registration. Coming from diverse backgrounds ranging from organizational work to evidence synthesis to policymaking, scholars Faduma Gure, Eric Linskens, and Christian Kershaw presented their proposals for new innovations or opportunities for improving the application and implementation of evidence-based medicine.

Faduma Gure, MSc, a knowledge translation and research specialist for the Association of Ontario Midwives, discussed the challenges of incorporating client perspectives to midwifery guidelines for the organization, which utilizes the GRADE approach. Pregnancy through the post-partum period is often a tumultuous time filled with decision-making, Gure explained, and more can be done to better understand the values and preferences of midwifery clients and to employ an equity lens when formulating recommendations. Gure proposed a solution that includes the development of an equity advisory group consisting of key stakeholders representative of Ontario’s population of midwifery clients. The organization could then involve these stakeholders through the entire guideline development process - from the initial setting of research priorities to the ultimate formulation of recommendations - and elicit important feedback about patient values and preferences as well as the potential impacts of a guideline on various communities.

Eric Linskens, BSc, serves the Minneapolis Veterans Affairs Evidence Synthesis Program and the Minnesota Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center. Linskens presented his current work applying the GRADE approach to existing systematic reviews which did not originally use GRADE. Linskens discussed some of the challenges that his team has faced as part of the initiative, as well as innovative solutions to these issues. For instance, a systematic review conducted by AHRQ may automatically rate down for inconsistency due simply to the inclusion of a sole study, whereas in the GRADE approach, this would not be the case. Additionally, an existing review may break down one clinical question into multiple smaller analyses of comparators or sub-populations, whereas it would be more clinically relevant to use these data to create one larger recommendation. To best solve these issues, Linskens noted, it is important to consider the end-user of any given review or guideline so that their needs can be best met. Additionally, transparently reporting all judgments around the analyses is key. Regarding his time at the workshop, Linskens said, “[i]t was very helpful to work through examples with the GRADE workshop facilitators in small group sessions. They answered our questions as they came up.”

Christian Kershaw, PhD, is a molecular neuroscientist who now works as a health policy analyst for CGS, a Medicare fee-for-service contractor. Dr. Kershaw used her personal experience transitioning from bench science to policymaking to inspire her presentation on the utility of cross-functional teams in medicine and healthcare policy. To develop a cross-functional team, Dr. Kershaw explained, it is best to identify a problem that would best be solved by a group of individuals with heterogeneous skills and backgrounds that would each uniquely serve a common goal or purpose. As an example, Kershaw discussed the development of a team to standardize the way information is used to form coverage decisions as part of the 21st Century Cures Act. The team is comprised of a medical doctor to understand the need for and content of the policies; an outreach and education specialist to understand their legal implications; and a basic research scientist to compile and assess the information. Leveraging individual team members’ strengths and encouraging innovation are keys to success when working in a cross-functional team. “I was impressed with the versatility of the GRADE framework,” Kershaw noted. “It was very informative to learn all of the different ways that the conference attendees were using GRADE to suit their projects.”

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Phoenix, AZ will be December 4, 2019.

Spring 2019 - Scholarship Recipients

Contributed by Madelin Siedler, 2018/2019 U.S. GRADE Network Research Fellow

Recently, we held the Tenth GRADE Guideline Development Workshop in Denver, Colorado. This workshop was one of the largest groups to date, with 51 participants traveling to the Mile-High City from as far away as Poland and Korea. During the workshop, participants focused on learning and applying the GRADE approach for diagnostic test accuracy. 

Two participants attended as recipients of the scholarship program funded by the U.S. GRADE Network and Evidence Foundation. This scholarship covers the cost of registration for workshop attendees who are newer to GRADE and have never attended a formal GRADE workshop. Scholars Janice Tufte and Dr. Irbaz bin Riaz presented on their innovative ideas for improving the development, implementation, or dissemination of guidelines with the aim of reducing bias in healthcare recommendations.

Scholarship recipients: Dr. Irbaz bin Riaz (L) and Ms. Janice Tufte (R), 

with scholarship coordinator, Dr. Shahnaz Sultan

Tufte, an independent consultant who leads patient-public partnership initiatives, presented on the unique opportunities of using patient partners during the development of GRADE guidelines. Patient partners are representatives of the patient population whom the guideline aims to serve. As part of a guideline panel, they offer fresh perspectives, ground the guideline development process with lived experience, and help the panel to identify and address differences in priorities among stakeholders.

Throughout her presentation, Tufte provided ways to improve how patient partners are involved in the guideline process, such as creating one-pagers and glossaries that cover the basics of GRADE methodology and inquiring beforehand about specific accommodations that might be needed in order to enhance the patient’s participation in the panel. “It was an honor to attend the GRADE Workshop in Denver as a Patient Partner Scholar,” said Tufte. “I felt like I was treated like a colleague where we were all learning together how to use GRADE tools to share best evidence within our individual systems and guidelines work.”

Dr. bin Riaz, an oncologist at Mayo Clinic, presented on a framework for developing living systematic reviews and guidelines to inform clinical decision-making, especially in topic areas undergoing rapid change. It can take several years for a systematic review and resulting clinical recommendations to be developed, Dr. bin Riaz explained. In the meantime, new drug approvals or indications, changes in drug labeling, or new information about potential risks and benefits of a treatment option can arise. As opposed to traditional, static documents, living systematic reviews and guidelines are continually updated as new evidence or important decision-making information comes to light. The ultimate goal of such an approach is to facilitate a more timely translation of medical knowledge into clinical practice, allowing patients and their providers to come to decisions informed by the totality of current evidence.

If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Orlando, Florida will be July 1, 2019.

Research Shorts: Borrowing of strength from indirect evidence

Contributed by M. Hassan Murad, M.D.

Network meta-analysis (NMA) is supposed to increase precision (because it includes more studies in the analysis); however, this is not always the case. This empirical study evaluated 915 possible treatment comparisons. The study used the recently proposed borrowing of strength statistic, which quantifies the percentage reduction in the uncertainty of the effect estimate when adding indirect evidence to an NMA

When only one study contributed direct evidence, NMAs resulted in reduced precision and no appreciable improvements in precision in 57.5% and 12.7% comparisons, respectively. When at least two studies contributed direct evidence, NMAs provided increased precision in 66.4% comparisons. The bottom-line is that, in sparse networks (i.e., networks that mostly consist of indirect evidence), NMA may not improve precision as much as stakeholders want and expect.

Reference: Lin L, Xing A, Kofler MJ, Murad MH. Borrowing of strength from indirect evidence in 40 network meta-analyses. Journal of clinical epidemiology. 2018 Oct 17.

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

Contributed by Madelin Siedler, 2018/2019 U.S. GRADE Network Research Fellow

Stating the “Obvious”: A Primer on Good Practice Statements in GRADE Guidelines

One of the benefits of the GRADE approach is that it provides a framework for the development of evidence-based recommendations that are clear and actionable for practicing clinicians even when only lower-quality evidence is available. However, in some particular instances, caution is warranted when developing a recommendation based on low-quality evidence or inference. “Good practice statements” are one such instance. 

The term “good practice statement” is sometimes interchanged for “motherhood statement”.” In either case, the practice being recommended is usually something that is already commonly accepted as beneficial or practical advice. It could even be seen as irrefutably “good” as motherhood and apple pie (hence the term). The nature of these types of statements is such that the action is seen as so obviously beneficial that it would be unduly onerous to conduct a review to demonstrate its efficacy.

An example of a good practice statement is the first recommendation from the American Gastroenterological Association (AGA)’s 2015 guideline on the management of asymptomatic pancreatic cysts, which reads, “The AGA recommends that before starting any pancreatic cyst surveillance program, patients should have a clear understanding of programmatic risks and benefits” (Vege et al., 2015).

How to Spot a Good Practice Statement
An easy way to identify a good practice statement is to restate the recommendation as its inverse: for instance, “patients should not have a clear understanding of programmatic risks and benefits.” If this “unstated alternative” is absurd or clearly does not conform to ethical norms, the original statement is likely a good practice statement (Guyatt et al., 2016).

The Problem with Good Practice Statements
The GRADE Working Group recommends that good practice statements be used sparingly, if at all. Because good practice guidance is typically based on several linked sources of indirect evidence, there is no way to tell whether the benefits of the proposed recommended action are as truly obvious or incontestable as they seem. And if they are (if the inverse of the proposed recommendation would be absurd or unethical) then they are likely unwarranted and can dilute the strength of the guideline as a whole. 

Sometimes, good practice statements may even appear as graded recommendations in a guideline (a decision that’s not recommended by the GRADE working group for the reasons above). In this case, the guideline authors may be tempted to make a strong recommendation based on low-quality evidence, which should ideally be a rare occurrence and based on well-defined criteria (Guyatt et al., 2015). 

Practical Advice for Dealing with Good Practice Statements
The GRADE Working Group recommends using the following checklist to determine whether a good practice statement is warranted:

1. Is the statement clear and actionable? 
2. Is the message really necessary in regards to actual health practice? 
3. After consideration of all relevant health outcomes and potential downstream consequences, will implementing the good practice statement result in large net positive consequences? 
4. Is collecting and summarizing the evidence a poor use of a guideline panel’s limited time and energy? 
5. Is there a well-documented clear and explicit rationale connecting the indirect evidence?

If the answer is 'yes' to all five questions, a good practice statement may be warranted for inclusion in a guideline document. When done correctly, good practice advice statements should appear as ungraded recommendations, meaning no formal rating of quality of evidence or strength of recommendation should be given (Guyatt et al., 2015).

However, many potential good practice statements will be eliminated through the use of this checklist. For instance, careful consideration of Question #3 could lead the panel to realize that the assumed net positive of a specific action may not be so obvious after all. In this case, the guideline panel should consider whether a thorough review of the evidence should be conducted and formal grading methods applied.

Research Shorts: When continuous outcomes are measured using different scales

Contributed by M. Hassan Murad, MD

Outcomes of great importance to patients, such as quality of life and severity of anxiety or depression, are often measured using different scales. When an outcome is measured using several scales across trials, it requires standardization to be pooled in a meta-analysis.

Common methods of standardization include using the standardized mean difference (SMD), converting continuous data to binary relative and absolute association measures, the minimally important difference (MID), the ratio of means, and transforming standardized effects back to original scales. The underlying assumption in all these methods is that the different scales measure the same construct. This paper, in BMJ, describes these methods and suggests approaches for interpretation.


Reference: Murad Mohammad Hassan, Wang Zhen, Chu Haitao, Lin Lifeng. When continuous outcomes are measured using different scales: guide for meta-analysis and interpretation BMJ 2019; 364 :k4817. https://www.bmj.com/content/364/bmj.k4817 

Fall 2018 Scholarship Recipients

Contributed by Madelin Siedler, 2018/2019 U.S GRADE Network Research Fellow

We were pleased to support the participation of three research scholars at our recent GRADE guideline development workshop held in Silver Spring, Maryland, October 17-19, 2018. By providing complementary registration, we hope that these scholars increased their understanding and application of the GRADE approach. As part of the U.S. GRADE Network/Evidence Foundation scholarship, recipients presented to fellow workshop participants about their interests and current endeavors in the field of guideline development.

Scholarship recipients: Oilvia Magwood, Mohamad Kalot, and Mohammed Alkhatib

A few details about our Fall 2018 scholarship recipients:

Mohamad Kalot, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in improving quality and decreasing disparities in healthcare through the development of evidence-based guidelines. Dr. Kalot presented on his current work in the development of guidelines for the management of rare diseases. Dr. Kalot explained how the development of such guidelines presents a unique challenge in that there is often a dearth of research on these populations, which can affect the directness of evidence among other factors.

“Since an important part of my conducted reviews and research deals with developing guidelines for rare diseases, I would ultimately face specific challenges in the process of assessing the certainty of evidence in my work, and I’m interested in exploring innovative methods to deal with these challenges,” said Kalot. “The GRADE workshop in Silver Spring didn’t only help me find solutions for my challenges, it made me see the research methodology world from a different, deeper and more practical perspective - especially with the very helpful tools [RevMan and GRADEpro Guideline Development Tool] that we learned about in the small groups sessions and the discussions about rating up and rating down the quality of evidence.”

Olivia Magwood, MPH, attended from the Buyere Research Institute in Ottawa, Ontario, where she has participated in the development of three national and international guidelines. Ms. Magwood presented on the development of the FACE (Feasibility, Acceptability, Cost, and Equity) Framework Stakeholder Survey. The FACE Stakeholder Survey aims to identify cognitive bias that may influence guideline development as well as inform the quality and impact and improve the uptake of evidence-based recommendations in various stakeholder groups.

“For me, the GRADE workshop highlighted the importance of building my network and having the right people on your team throughout the guideline development process,” explained Magwood. “This includes involving methodologists early and throughout guideline development, as well as emphasizing patient perspectives, especially while deciding on patient-important outcomes. Guideline development is truly a collaborative process that benefits from multi-stakeholder engagement.”

Mohammed Alkhatib, MD, a postdoctoral research fellow at University of Kansas Medical Center, Kansas City, Kansas, applied for the scholarship due to his interest in examining and refining the adoption, adaptation and de novo development (or “adolopment,” for short) of guidelines to be used in lower-resource settings. Dr. Alkhatib presented on the importance of this approach, as well as the unique challenges of tailoring the development of clinical recommendations to settings with limited resources, such as developing nations and areas of conflict.

“It was an exceptional experience for me to present my proposal in front of high-level scholars of guideline development and to listen to their positive feedback,” said Alkhatib. “[The] GRADE workshop was very helpful for me at the level of interpretation of [systematic reviews] and how to judge strengths and weaknesses using GRADEpro.”

**If interested in applying for a scholarship to future GRADE workshops, more details can be found here: https://evidencefoundation.org/scholarships.html. Please note the deadline for applications to our next workshop in Denver, Colorado will be January 1, 2019.