U.S. GRADE Network Describes Experience Moving to All-Virtual Workshop Format

On March 4-6, 2020, the U.S. GRADE Network held an in-person workshop in Phoenix, Arizona, much like any of the 11 workshops to have come before it. Participants and facilitators enjoyed a taco buffet bar together at the first night's reception, sat together in small and large rooms to learn and collaborate, and mingled over coffee and pastry refreshments during breaks.

One week later, the World Health Organization announced that COVID-19 had reached pandemic proportions. 

Over that summer, the USGN took our workshops online, hosting three consecutive fully virtual workshops in October 2020, May 2021, and October 2021. While some changes were made (the addition of multiple, 45-60 minute breaks, for instance, to accommodate eating times in multiple time zones), much of what lay at the heart of a GRADE workshop remained: a three-day format  including plenary lectures from PICOs to recommendations, presentations by Evidence Foundation scholars, and small-group, hands-on experiential learning opportunities.

The USGN's shift to an all-virtual setting, and its challenges as well as opportunities for growth, are presented in a new paper by Siedler and colleagues published online in the BMJ Evidence-Based Medicine journal. Using routine feedback survey data collected both before and after the pandemic, the authors (all GRADE workshop facilitators) found that...

• Perceived understanding of GRADE improved to the same extent in virtual and in-person formats,
• At least half of attendees (54-62%) indicated that the virtual format was important for their ability to attend, and
• Participants indicated a high degree of workshop satisfaction and perceived educational value. Similar results were observed for the level of knowledgeability of speakers, value of plenary sessions, and helpfulness of small-group sessions.

The major takeaway from the USGN's experience in an all-virtual format is that, based upon positive feedback and the ability to reach a global audience of learners, it will continue to offer learning opportunities in a virtual setting this year and beyond.

In fact, the next all-virtual workshop will take place November 30-December 2, 2022, and registration is now open at www.gradeconf.org.

Siedler MR, Murad MH, Morgan RL, et al. (2022). Proof of concept: All-virtual guideline development workshops using GRADE during the COVID-19 pandemic. BMJ Evidence-Based Medicine (online before print). Manuscript available from publisher's website here.

In Studies of Patients at High Risk of Death, More Explicit Reporting of Functional Outcomes is Needed

Randomized controlled trials examining the effects of an intervention in patients with a high risk of death will often also include functional outcomes - such as quality of life, cognition, or physical disability. However, the death of patients before these outcomes can be assessed (also known as "truncation due to death") can confound the results of a "survivors-only" analysis, especially if mortality rates are higher in certain groups than others. 

A new methodology review of studies published within 5 high-impact general medical journals from 2014 to 2019 provides insight into this phenomenon and suggestions for improving how functional outcomes are handled. To be eligible for the review, a study needed to be a randomized controlled trial (RCT) with a mortality rate of at least 10% in one arm and to report at least one functional outcome in addition to mortality. The authors recorded the outcomes analyzed, the type of statistical analyses used, and the sample population of each of the 434 included studies. For most (351, or 79%) of these, function was a secondary outcome, while it was a primary outcome for 91 (21%) of them.

Only one-quarter (25%) of the functional outcomes within the studies that examined them as secondary outcomes used an approach that included all randomized patients (intention-to-treat); for the studies for which functional outcomes were the primary outcomes analyzed, this proportion was 60%.

The authors provide suggestions for best ways to handle and report data in these studies:

• In the methods rather than only in tables or supplementary material, explicitly state the sample population from which the functional outcomes were drawn, whether it's survivors-only or another type of analysis.
• If a survivors-only analysis is used, the authors should report the baseline characteristics between the groups analyzed and transparently discuss this as a limitation within the discussion section.
• If all randomized participants are analyzed regardless of mortality, authors should report the assumptions upon which these analyses are based; for instance, if death is one outcome ranked among others in a worst-rank analysis, the justification for the ranking of outcomes should be discussed in the methods, and the implications of these decisions included in the discussion section. 

Colantuoni E, Li X, Hashem MD et al. (2021). A structured methodology review showed analyses of functional outcomes are frequently limited to "survivors only" in trials enrolling patients at high risk of death. J Clin Epidemiol (e-pub ahead of print).

Manuscript available here.

Equity Harms Related to Covid-19 Policies: Slowing the Spread Without Increasing Inequity

Since COVID-19 was first declared a pandemic in March of this year, numerous policies around the world have implemented some degree of lockdown, slashing social events and gatherings, shuttering once-bustling businesses and changing the face of the global economy. While the lockdowns in place were likely necessary to reduce the infection rate and resulting morbidity and mortality associated with the coronavirus, there are potentially undesirable consequences of these policies that affect measures of equity. In a new publication, Glover and colleagues present a framework for considering these effects and weighing them against the benefits of slowing the spread.

The work builds off of a novel combination of two existing frameworks. First, the Lorenc and Oliver framework lays out five potential harms of public health interventions which require mitigation: direct health harms, psychological harms, equity harms, group and social harms, and opportunity costs. Second, the PROGRESS-Plus health equity framework provides a list of 11 general categories that can affect measures of equity: Place of residence, Race, Occupation, Gender/sex, Religion, Education, Socioeconomic status, Social Capital, sexual orientation, age, and disability. Each of the two frameworks' individual components are used as a lens to examine the other. The resulting matrix of 55 potential sources of inequity related to the COVID-19 pandemic and its resulting public health policies provides an exemplary approach to considering all aspects of any large-scale public health intervention and the impact its implementation may have on inequity.

Key to the authors' resulting framework is the concept that both the policy responses to the pandemic and the nature of the pandemic itself are potential sources of inequity. For instance, individuals in lower-income occupations are also typically considered essential workers, and are less likely to have a safety net that would allow them to choose not to work or a job that is compatible with working remotely. Thus, the existing systemic inequities are exacerbated by the fact that they are now more likely to be exposed to the virus by continuing to go to work outside the home. However, policymakers can help reduce the impact of their policies on these sources of inequity - as well as ones caused by lockdown policies more directly - by considering mitigation strategies when implementing these policies (for example, by mandating improved sanitation, personal protective equipment, and social distancing for workers in vulnerable occupations). The figure below from the publication provides an overview of the relationship between the pandemic, policy responses and their resulting inequities, and potential points of intervention.

Click to enlarge.

The framework also allows for a more nuanced consideration of context in efforts to reduce the spread of coronavirus. Policies that are highly effective and viable in higher-income countries or areas with greater population density, for instance, may not be as beneficial in low- and middle-income countries and may even result in greater inequity. As with any intervention of any scale, the potential harms must be weighed against the desirable effects, and the context of the given intervention is key. This framework allows for consideration of a wider range of impacts when attempting to reduce illness and mortality in the age of a pandemic.

Glover, R.E., van Schalkwyk, M.C.I., Akl, E.A., Kristjannson, E., Lofti, T., Petkovic, J., ... & Welch, V. (2020). A framework for identifying and mitigating the equity harms of COVID-19 policy interventions. J Clin Epidemiol 128:35-48.

Manuscript available from the publisher's website here. 

Rapid Guidelines in GRADE Pt. III: A checklist for rigorously rapid recommendations

In recent posts, we have introduced the concept of rapid recommendations as well as how developers of these recommendations at the World Health Organization (WHO) perceive facilitators and barriers to this process. This information was gathered as part of a published series on rapid guidance in 2018.

In the final part of the series, Morgan and colleagues propose an extension of the G-I-N/McMaster Checklist for Guideline Development aimed at those producing rapid guidelines. Comprising 21 discrete guideline principles that align with the original Guideline Development Checklist, the checklist is a tool for developers to take stock of the resources available for their rapid guideline development goals and identify areas in need of improvement. Important considerations include:

·      Make use of virtual meetings (Principle 7) and pre-meeting voting (Principle 15) to expedite the drafting of recommendations.

·      If possible, limit guideline panel composition to those not reporting financial COIs, or if not possible to provide necessary topic expertise, transparently declare any modifications of existing COI policies for the topic at hand (Principle 9).

·      Limiting the number of PICOs (Principle 10) and limiting the assessment of outcomes to only those deemed critical (Principle 11).

·      Consider ways to facilitate systematic review stage, such as updating existing reviews, developing rapid reviews, or tailoring search criteria to a smaller scope (Principle 13)

·      Arrange for external reviewers early on in the process so that they are quickly deployed when a draft is available for review (Principle 18).

The full checklist extension for rapid guidelines can be viewed here.

Morgan, R.L., Florez, I., Falavigna, M. et al. Development of rapid guidelines: 3. GIN-McMaster Guideline Development Checklist extension for rapid recommendations. Health Res Policy Sys 16, 63 (2018). https://doi.org/10.1186/s12961-018-0330-0

Manuscript available at the publisher's website here.

Rapid Guidelines in GRADE Pt. II: Rapid Recs in the Real World

In Part I of our series on rapid guidelines, we discussed the utility and terminology of rapid recommendations: those recommendations made in response to an urgent public health issue with timeframes ranging from a few short hours up to three months' time.

Who develops rapid guidelines?

In the first of a three-part series on rapid guidance published in 2018, Kowalski and colleagues conducted a systematic survey of the methodologies and processes of rapid guideline-producing organizations. Nomenclature used to identify these documents varied by organization, from “rapid advice guideline” to “interim guidance” to “short clinical guideline.” While the quality of these documents as assessed with the AGREE II tool was variable, it was greater in documents from the WHO and NICE than it was for the CDC or other smaller organizations. While NICE guidelines were of higher quality as assessed with the domains of AGREE II, they took substantially more time to develop than those from WHO.

It's important to note that while terminology differs between organizations, the word "interim" has been used to connote a response to an emergent public health issue with shorter time frames than a rapid guideline - typically on the order of 1-3 weeks. 

Organization	Nomenclature	AGREE II domain score range (lowest – highest)	Development timeline (from manual)
WHO	Rapid advice guideline	54 - 92	1-3 months
NICE	Short clinical guideline	81 - 94	11-13 months
CDC	Interim guidance	10 - 82	Not reported
Other	Interim guidelines, interim position statement, clinical guidelines	21 - 67	Not reported

Common Challenges and Facilitators to Rapid Guideline Development

While both the World Health Organization (WHO) and the National Institute for Health and Care Excellence (NICE) reported the use of a systematic review of the evidence to guide recommendations, common issues among developers included a lack of reporting on the management of conflict of interest, external review, and of the process for the drafting of recommendation.

In follow-up qualitative interviews with guideline-developing staff from WHO, participants cited a lack of adequate staffing, monetary resources, and evidence as key obstacles to the development of rapid guidelines. While the development of a systematic review is likely one of the more time-consuming elements of a rapid guideline process, most participants agreed that it is a fundamental part of developing trustworthy guidance that should not be skipped if possible. 

Participants also indicated that the external/peer review process can add unwanted time to the development of rapid guidelines. To this effect, developers can consider limiting the reach of peer review to the final draft only as well as reducing the ability to drastically change recommendations in a way that would require a reconvention of the guideline panel. Virtual conferencing technology was named as a facilitator to developing guidelines on a quicker schedule by reducing the need for face-to-face meetings. 

For a checklist to guide the development of rapid recommendations, see the G-I-N/McMaster checklist extension for rapid guidelines.


Kowalski, S.C., Morgan, R.L., Falavigna, M. et al. Development of rapid guidelines: 1. Systematic survey of current practices and methods. Health Res Policy Sys 16, 61 (2018).

Manuscript available at the publisher's website here.  

Florez, I.D., Morgan, R.L., Falavigna, M. et al. Development of rapid guidelines: 2. A qualitative study with WHO guideline developers. Health Res Policy Sys 16, 62 (2018).

Manuscript available at the publisher's website here.