COVID-END Working Groups Call for Living Systematic Reviews and Considerations of Health Equity in Evidence Synthesis and Guideline Efforts

The COVID-19 pandemic was not only associated with a rapid worldwide spread of a virus, but also of large amounts of information across the globe - not all of which was trustworthy or credible. Experts call this an "infodemic." In order to improve the synthesis and dissemination of trustworthy information in a manner that could keep up with the fast pace and ever-changing landscape of knowledge on COVID-19, the COVID-19 Evidence Network to support Decision-making (COVID-END) was established.

In a paper published in this month's issue of Journal of Clinical Epidemiology, McCaul and colleagues describe how the COVID-19 pandemic ushered in an urgent need to rapidly understand the etiology and management strategies for the disease, and to disseminate this information far and wide. However, a lack of collaboration resulting in duplication of work across institutions and countries hampered these efforts. COVID-END, comprising two working groups  dedicated to overseeing the coordination and dissemination of trustworthy evidence syntheses and guidelines, was a result of these unprecedented needs. The effort also included an Equity Task Group that evaluated the impact of evidence synthesis and recommendations on matters of health and socioeconomic disparities arising from or exacerbated by the pandemic.

Figure from McCaul et al. describing the efforts of COVID-END

The goal of the project, in the authors' words, was to support the "evidence supply side" by promoting already available resources and work led by institutions across the globe, both for those involved in evidence synthesis or the formulation of recommendations based on the evidence. The avoidance of effort duplication was highlighted by, for instance, urging guideline developers to first search for existing high-quality and up-to-date guidelines before beginning work on new recommendations. The development and use of living systematic reviews, which are continually updated as new evidence becomes available, is further highlighted as a way to improve the timeliness of evidence syntheses while reducing efforts put into new projects.

McCaul, M., Tovey, D., Young, T., et al. (2022). Resources supporting trustworthy, rapid and equitable evidence synthesis and guideline development: Results from the COVID-19 Evidence Network to support Decision-making (COVID-END). J Clin Epidemiol 151: 88-95. Manuscript available at publisher's website here. 

8 Steps Toward Incorporating Equity in Rapid Reviews and Guidelines

Along with the mass mobilization of systematic reviews of evidence ushered in by the COVID-19 pandemic was also the need to synthesize evidence and disseminate results as rapidly as possible. As part of the process of formulating guidelines based upon rapid reviews, the impact of decisions and policies on equity should be considered. In a newly published paper, Dewidar and colleagues provide specific steps for incorporating stakeholders and improving the consideration of equity in the context of rapid guidelines.

The project was part of work conducted by the Equity Task Force of the global COVID-19 Evidence Network to support Decision-making COVID-END) network. The team was diverse in terms of the gender (70% women), regions (17% from low-middle-income countries), and career stages (40% early career) represented. The resulting guidance was created in line with the steps outlined in the Cochrane Handbook's chapter on equity and followed the PRISMA-Equity (PRISMA-E) extension for reporting. The team then identified published systematic reviews related to COVID-19 that focused on populations experiencing inequities as categorized by the PROGRESS-Plus paradigm - for instance, by Place of Residence (health systems in rural areas and their preparedness for outbreaks), Education (the impact of educational attainment on adherence to COVID-19 health guidelines), and Disability (the impact of COVID-19 on those with disabilities) - for examples that review authors can incorporate equity into their own reviews.

The authors conclude that greater involvement of diverse stakeholders can encourage the consideration of more diverse social factors in the development and interpretation of systematic reviews and resulting guidelines and policies. Rapid reviews also benefit from having a translation plan that includes methods for disseminating findings in a way that is consistent with the goal of reducing inequities. 

Dewidar, O., Kawala, B.A., Antequera, A., et al. (2022). Methodological guidance for incorporating equity when informing rapid-policy and guideline development. J Clin Epidemiol 150: 142-153. Manuscript available at the publisher's website here. 

U.S. GRADE Network Describes Experience Moving to All-Virtual Workshop Format

On March 4-6, 2020, the U.S. GRADE Network held an in-person workshop in Phoenix, Arizona, much like any of the 11 workshops to have come before it. Participants and facilitators enjoyed a taco buffet bar together at the first night's reception, sat together in small and large rooms to learn and collaborate, and mingled over coffee and pastry refreshments during breaks.

One week later, the World Health Organization announced that COVID-19 had reached pandemic proportions. 

Over that summer, the USGN took our workshops online, hosting three consecutive fully virtual workshops in October 2020, May 2021, and October 2021. While some changes were made (the addition of multiple, 45-60 minute breaks, for instance, to accommodate eating times in multiple time zones), much of what lay at the heart of a GRADE workshop remained: a three-day format  including plenary lectures from PICOs to recommendations, presentations by Evidence Foundation scholars, and small-group, hands-on experiential learning opportunities.

The USGN's shift to an all-virtual setting, and its challenges as well as opportunities for growth, are presented in a new paper by Siedler and colleagues published online in the BMJ Evidence-Based Medicine journal. Using routine feedback survey data collected both before and after the pandemic, the authors (all GRADE workshop facilitators) found that...

• Perceived understanding of GRADE improved to the same extent in virtual and in-person formats,
• At least half of attendees (54-62%) indicated that the virtual format was important for their ability to attend, and
• Participants indicated a high degree of workshop satisfaction and perceived educational value. Similar results were observed for the level of knowledgeability of speakers, value of plenary sessions, and helpfulness of small-group sessions.

The major takeaway from the USGN's experience in an all-virtual format is that, based upon positive feedback and the ability to reach a global audience of learners, it will continue to offer learning opportunities in a virtual setting this year and beyond.

In fact, the next all-virtual workshop will take place November 30-December 2, 2022, and registration is now open at www.gradeconf.org.

Siedler MR, Murad MH, Morgan RL, et al. (2022). Proof of concept: All-virtual guideline development workshops using GRADE during the COVID-19 pandemic. BMJ Evidence-Based Medicine (online before print). Manuscript available from publisher's website here.

Use of an Evidence-to-Decision Framework is Associated with Better Reporting, More Thorough Consideration of Recommendations

In the guideline development process, a panel should use a defined framework to consider multiple aspects of a clinical decision, including but not limited to the certainty of the underlying evidence, potential impact on resource use, or variability in the values and preferences of patients and other stakeholders. Such frameworks include the GRADE Evidence-to-Decision (EtD) format as well as others such as the "decision-making triangle" and Guidance for Priority-Setting in Health care (GPS Health). 

To better understand the prevalence and use of these various frameworks within guidelines, Meneses-Echavez and colleagues systematically searched for guidelines and related guideline production manuals published between 2003 and May 2020. Items were screened and extracted by two independent authors, with a total of 68 full text documents included and analyzed.

Of these documents, most (93%) reported using a structured framework to assess the certainty of evidence, about half (53%) of which used GRADE or adapted systems based on GRADE (10%). Similarly, 88% of documents reported using a framework to rate the strength of recommendations, with about half (51%) using the GRADE approach. However,  only about two-thirds (66%) of the included documents explicitly stated the process for formulating resulting recommendations. 

Finally, the GRADE framework  was most commonly used for the evidence-to-decision, being cited in 42% of the included articles, with other reported frameworks including NICE (8%), SIGN (8%) and USPSTF (4%). Articles using the GRADE EtD framework reported considering more criteria than those using alternative approaches. The most commonly used criteria across documents included desirable effects (72%), undesirable effects (73%), and the certainty of evidence of effects (73%); the least commonly applied criteria were acceptability (28%), certainty of the evidence of required resources (25%), and equity (16%). 

The use of any EtD framework was associated with a greater likelihood of incorporating perspectives (odds ratio: 2.8; from 0.6-13.8) and subgroup considerations (odds ratio:7.2; from 0.9-57.9), as was the use of GRADE compared to other EtDs (odds ratios: 1.4 and 8.4). These differences also affected whether justifications were reported for each judgment as well as the inclusion of notes to consider for the implementation of recommendations and for monitoring and evaluating recommendations.  

The authors conclude that guidance documents stand to benefit from the more explicit reporting of how recommendations are formulated, from the initial grading of the certainty of underlying evidence to the consideration of how recommendations will affect various criteria such as resource use and equity. These changes, in the words of the authors, may "enhance transparency and credibility, enabling end users to determine how much confidence they can have in the recommendations; facilitate later adaptation to contexts other than the ones in which they were originally developed; and improve usability and communicability of the EtD frameworks." 

Meneses-Echaves JF, Bidonde J, Yepes-Nuñez JJ, et al. (2022).  Evidence to decision frameworks enabled structured and explicit development of healthcare recommendations. J Clin Epidemiol 150:51-62. Manuscript available at publisher's website here.

Only One-Third of a Sample of RCTs Had Made Protocols Publicly Available, New Report Finds

Earlier this year, a study in PloS Medicine found that nearly one-third (30%) of a sample of randomized controlled trials (RCTs) had been discontinued prematurely, a number that had not improved over the previous decade. Furthermore, for every 10% increase in adherence to SPIRIT protocol reporting guidelines, RCTs were 29% less likely to go unpublished (OR: 0.71; 95% confidence interval: from 0.55 to 0.92), and only about 1 in every 5 unpublished trials had been registered.

Now, in this month's issue of Journal of Clinical Epidemiology, Schönenberger and colleagues have released a study of the availability of RCT protocols from a sample of published works.

Public availability of study protocols, the authors argue, improves research quality by promoting thoughtfulness in methodological design, reducing selective outcomes reporting or "cherry-picking," and reducing the misreporting of results while promoting ethical compliance. This is especially the case when trial protocols are made available before the publication of study results. 

From a random sample of RCTs approved by ethics committees in Switzerland, Germany, Canada, and the United Kingdom in 2012, the authors examined the proportion of studies that had publicly available protocols and the nature of how the protocols were cited and disseminated. Of the resulting 326 RCTs, 118 (36.2%) had publicly available protocols. Of the protocols, nearly half (47.5%) were available as standalone peer-reviewed publications while 40.7% were available as supplementary material with the published results. A smaller proportion (10.2%) of protocols were available on a trial registry. 

Studies with a sample size of >500 or that were investigator- (non-industry)-sponsored were more likely to have publicly available protocols. The nature of the intervention (drug versus non-drug) did not appear to affect protocol availability, nor did whether the trial was conducted in a multicenter or single-center setting. The majority (91.8%) of protocols were made available after the enrollment of the first patient, and just 2.7% were made available after publication of trial results. Protocols were commonly published shortly before the trial results, at a median of 90% of the time between the start of the trial and its publication.

As this sample comprised only RCTs published in 2012 and by relatively high-income countries, it is unclear whether public protocol availability has improved over time or may be different in other global regions. However, the authors argue, these numbers lend credence to the need for efforts to improve the public availability of RCT protocols, such as through trial registries or requirements by publishing or funding bodies.

Schönenberger, C.M., Griessbach, A., Heravi, A.T., et al. (2022). A meta-research study of randomized controlled trials found infrequent and delayed availability of protocols. J Clin Epidemiol 149:45-52. Manuscript available at publisher's website here. 

Second USGN Systematic Review Workshop Features a Record Nine Scholars From Around the Globe

Earlier this month, the U.S. GRADE Network held its second two-day, comprehensive, all-virtual systematic review workshop. This workshop allowed participants to learn about each step of the systematic review process, from designing a search strategy to meta-analysis to preparing a manuscript for publication. True to USGN style, the workshops consisted of a mixture of large-group lectures and smaller-group experiential learning components in which participants received a tutorial of Rayyan (a free online screening tool), assessed studies for risk of bias, and created risk of bias and forest plots in Cochrane's Review Manager software.

Uniquely, this workshop featured nine Evidence Foundation scholars who attended the workshop free of charge. As part of their applications for this scholarship, these participants described a proposed or current systematic review project related to health care, with a preference given to projects aimed at addressing inequities or with a focus on underserved populations. The nine accepted applicants provided a diverse array of exciting projects - from HIV pre-exposure prophylaxis to interventions for rabies control to weight-bearing exercise in pregnant patients - and hailed from across the globe, from Canada and Benin to Turkey, Syria, Dubai, and Kazakhstan.

Be the first to hear about these and other trainings @USGRADEnet on twitter or at www.systematicreview.org.

Note: applications for scholarships to attend the upcoming GRADE Guideline Development Workshop, held virtually November 30 - December 2, 2022, close September 30. See application details here.

Standardized Mean Difference Estimates Can Vary Widely Depending on the Methods Used, New Review Finds

In meta-analyses of outcomes that utilize multiple scales of measurement, a standardized mean difference (SMD) may be used. Randomized controlled trials may also use SMDs to help interpret the effect size for readers. Most commonly, the SMD reports the effect size with Cohen's d, a metric of how many standard deviations are contained in the mean difference within or between groups (e.g., an intervention caused the outcome to increase or decrease by x number of standard deviations, or the two groups were x number of standard deviations different from one another with regards to the outcome). This is typically done by dividing the difference between groups, or from pretest to posttest in a single group, by some form of standard deviation (e.g., pooled standard deviation at baseline, posttest, or the standard deviation of change scores). Cohen's d is often utilized because a general rule of interpretation has been suggested: 0.2 is a small effect, 0.5 is a medium-sized effect, and 0.8 is large.

However, there are multiple ways to approach the calculation of SMDs, and these may result in varying interpretations of the size of the effect. To further investigate this, Luo and colleagues recently published a review of 161 articles using SMDs and the way they can be calculated. Of the 161 randomized controlled trials published since 2000 and reporting outcomes with some form of SMD, the authors calculated potential between-group SMDs using reported data and up to seven different methodological approaches.

Some studies reported more than one type of SMD, meaning that 171 total SMD approaches were reported across the 161 studies. Of these, 34 (19.9%) did not describe the chosen method at all, 84 (49.1%) reported but in insufficient detail, and 53 (31%) reported the approach in sufficient detail. The confidence interval was only reported for 52 (30.4%) of SMDs. Of the 161 individual articles, the rule for interpretation was clearly stated in only 28 (17.4%). 

The most common method of calculating SMD was using a standard deviation of baseline scores, seen in 70 (40.9%) of studies. Meanwhile, 30 (17.5%) used posttest standard deviations and 43 (25.1%) used the standard deviation of change scores.

Figure displaying the variability of SMD estimates across 161 included studies. Click to enlarge.

Of all the potential ways to calculate SMD, the median article varied by 0.3 - which could potentially be the difference between a "small" and "moderate" or "between a "moderate" and "large effect size for Cohen's d using Cohen's suggested rule of thumb. The studies with the largest variation tended to have smaller sample sizes and greater reported effect sizes.

This work raises an important point, which is that while no one method for the calculation of SMDs is considered superior to another, if calculation approaches are not prespecified by researchers, different methods could be tried until the most impressive effect size is reached. To help prevent these issues, the authors suggest prespecifying the analytical approach and reporting SMDs together with raw mean differences and standard deviations to further aid interpretation and provide context. 

Luo, Y., Funada, S., Yoshida, K., et al. (2022). Large variation existed in standardized mean difference estimates using different calculation methods in clinical trials. J Clin Epidemiol 149: 89-97. Manuscript available at the publisher's website here.