Three's a Crowd: How to Deal with More than Two Arms in a Meta-Analysis

It is not uncommon to come across the following scenario: when conducting a meta-analysis between two arms (e.g., an active therapy vs. a placebo), the meta-analyst includes a study that actually included two active arms (e.g., two different doses of the same experimental drug vs. placebo, two different routes of administration, etc.) Let's say that both of these arms were relevant to the clinical question. How should meta-analysis be undertaken in this case? A new tutorial article published in Cochrane Evidence Synthesis and Methods gives a primer on how to approach this common conundrum. 

Including the study twice in the forest plot – for instance, with one dose versus placebo and the other versus the same placebo group – is statistically problematic. It leads to a "unit of analysis" error by essentially "double-counting" the participants in the control group and violating the assumption that every individual participant is only counted twice. (Aside: this is also a common error in meta-analyses combining multiple similar outcomes – e.g., including the handgrip strength of both the dominant and non-dominant hand in the same forest plot – and risks committing the same violation unless advanced multi-level statistical techniques are used to account for this). 

This leaves two basic options for including the data from more than two study arms into the same forest plot: combining interventions that are similar, and splitting the control group in half. For instance, if the three groups in question are as such (assuming a dichotomous outcome):

• Experimental group A: 50 participants, 45 of whom had the event.
• Experimental group B: 50 participants, 41 of whom had the event.
• Control group: 50 participants, 22 of whom had the event.

These two experimental groups can either be combined (100 participants, 89 of whom had the event) and compared to the control group as-is, or they can be split out and compared (on separate lines of the forest plot) to half of the control group on each line:

• Experimental group A (45 out of 50) versus control (11 out of 25)
• Experimental group B (41 out of 50) versus control (11 out of 25).

Both approaches will yield very similar pooled results.

In the case of a continuous outcome, the same general approaches can be applied. However, if pooling two or more arms together, a pooled mean and SD will need to be calculated using the following equations from the Cochrane Handbook:

Reference: Axon, E., Dwan, K., & Richardson, R. (2023) Multiarm studies and how to handle them in a meta-analysis: A tutorial. Cochrane Evidence Synthesis and Methods. Available at publisher's website here.

Review of Time-to-Event Outcomes Analyzed in 50 Systematic Reviews Indicates Lack of Rigorous Reporting

Commonly used in the fields of oncology, cardiology, and others, time-to-event (TTE) outcomes assess not only the occurrence of an event but the amount of time that has lapsed leading up to its occurrence. For individuals in which the event did not occur, their time under observation is included.

TTE outcomes can provide useful insight into, for instance, into how long individuals survive when taking a new drug for advanced cancer. Statistical methods of calculating TTE outcomes include the use of curves and probabilities (Kaplan-Meier curves) and hazard ratios (HRs). 

In addition to being more statistically complex than a simple risk ratio, TTE outcomes are often not reported adequately enough in trials to be used in a meta-analysis without some amount of imputation or exclusion of data, which can introduce error.

In an article published in the July 2023 issue of Journal of Clinical Epidemiology, Goldkuhle and colleagues more closely examined the use of TTE outcomes in meta-analyses included in both Cochrane and non-Cochrane systematic reviews between 2017 and 2020. In the 50 included reviews, a median two TTE outcomes were included in meta-analyses, the most common being comparing the use of biologics and drugs in the treatment of neoplasms. 

However, a lack of clear reporting in the 235 trials informing these systematic reviews could easily lead to incomplete data. For instance, only 82% of the trials included a measure of follow-up duration. Information about missing data was only reported for each trial arm in 134 (57%) of the trials, and about one-third of trials reported no information at all in this respect. 

The authors conclude that trial authors using TTE outcomes should more stridently follow the Consolidated Standards of Reporting Trials (CONSORT) extension to trial outcomes, while specific guidance is needed for the reporting of meta-analysis of TTE outcomes. 

Goldkuhle, M. et al. (2023). Meta-epidemiological review identified variable reporting and handling of time-to-event analyses in publications of trials included in meta-analyses of systematic reviews. J Clin Epidemiol 159: 174-189.

The full-text publication can be accessed here.

McMaster and Guidelines International Network launch Collaboration Toolkit for Guideline Developers

Collaboration between medical professional societies and health organizations can increase the efficiency of development efforts, reduce waste, and help share resources, knowledge, and skills within and across disciplines. However, an international needs assessment of guideline developers found that certain aspects of collaboration were specifically challenging, including difficulties reconciling differences in guideline methodology and the time required for establishing collaboration agreements.

Since then, McMaster University and the Guidelines International Network (G-I-N) have worked on developing a toolkit for organizations looking to inspire, guide, and facilitate their own collaborations. The toolkit, launched at this year's G-I-N conference in Glasgow, Scotland, includes access to helpful resources such as a Memorandum of Understanding template for establishing partnerships and a glossary for making sure all parties are on the same page regarding important terms. A list of collaboration-related publications offers further knowledge and guidance for aspiring collaborators.

Check out the toolkit here and stay posted for update and additions!

EF Scholars Talk High-Speed Ways to Apply Evidence to Practice at Austin GRADE Workshop

 At the most recent GRADE Guideline Development Workshop held in Austin, Texas, our team was joined by two impressive scholars making moves in the field of evidence synthesis and evidence-based medicine. These scholars, Drs. Syed Arsalan Naqvi and Juan Ruiz, were selected to have their workshop registration fee waived based on their application materials, including a description of a current or proposed project to reduce bias in healthcare. They also had the opportunity to present on these topics and field questions from the other workshop attendees.

Syed Arsalan Ahmed Naqvi, M.B.B.S., presented on a current project to apply machine learning to develop an algorithm to automatically assess certainty of evidence in living systematic reviews within a living interactive evidence (LivE) framework. Certainty of evidence assessment requires "repetitive evaluations whenever new studies or updated reports of previously included studies are incorporated into the existing body of evidence," Dr. Naqvi explained. "The manual efforts required to assess certainty of evidence necessitate considerable time and resources, which become even more cumbersome with periodic updates." Dr. Naqvi's project described the proposal for "an adjudication system which leverages an interactive web-based graphical user interface and rule-based algorithms" for assessing certainty in the context of living systematic reviews of randomized controlled trials.

Of his experience at the workshop, Dr. Naqvi said, "“The GRADE guideline workshop was an enriching experience. It was well organized, and precise, focusing on the most important aspects of clinical guideline development. I really enjoyed the small group sessions with the faculty and the participants."

Dr. Juan Ruiz presented his proposal to implement a rapid review and question-and-answer program for hospitalists applying evidence-based medicine in everyday practice. "Doctors have questions every day at the point of care that are not always [addressed], Dr. Ruiz explained. "Factors associated with unanswered medical questions" include "lack of time" and "lack of knowledge and skills to apply evidence-based medicine tools." By training onco-hospitalists in GRADE and creating an accessible database with clinical questions, answers, and recommendations based on evidence-based methodologies, Ruiz hopes to help bridge this gap in the evidence-to-practice pipeline.

"I had the opportunity to participate as a Scholar of the Evidence Foundation in the US GRADE Network workshop in Austin, TX in May 2023," said Dr. Ruiz. "This workshop was focused on non-RCTs, an area of great interest to me since I work analyzing large national databases and electronic medical records of cancer patients. I presented and received valuable feedback for a project to be implemented in a cancer hospital to develop recommendations for daily practice using the GRADE approach. It was a very constructive three-day experience where I was able to learn new skills and share knowledge."

Interested in becoming a scholar? Applications to attend our next fully virtual fall workshop, held October 25-27, 2023, for free close on August 31. See application details here. 

GRADE Guideline Development Workshop - now with INGUIDE!

We at the U.S. GRADE Network are excited to announce that, for the first time ever, we have partnered with INGUIDE, a credentialing system for guideline developers, to offer free access to their Level 1 online training to participants of our Guideline Development Workshop and Guideline Fundamentals for Panel Members and Organization Staff pre-workshop course. 

This announcement comes just in time for our next workshop, held in Austin, Texas May 17-19, for which there are still spots available at our workshop website.

The INGUIDE program, the result of a collaboration between the Guidelines International Network (G-I-N) and McMaster University, provides certification of individuals involved in guideline development on four successive levels, from a Guideline Panel Member to becoming a Guideline Development Credentialing Instructor themselves. 

To obtain access to the INGUIDE Level 1 course, simply register for and attend our 3-day GRADE guideline development workshop in addition to the pre-workshop course on Guideline Fundamentals for Panel Members and Organization Staff. This 3-hour pre-workshop course will cover all of the necessary information about the practical aspects of guideline development, including:

• the importance of trustworthy guidelines
• checklists for the development of guidelines
• the structure and roles of a guideline panel
• the process of grading evidence and formulating recommendations

The main 3-day workshop will cover important aspects of assessing risk of bias, rating the certainty of evidence, and considering other important factors when using GRADE to develop a guideline.

Our pre-workshop course combined with our main three-day in-person course will provide all of the training required for a breeze-through experience online. You will receive free access to the ~2-3 hour online course, comprising four modules, that you can complete at your convenience after the workshop. Upon completion, you will be officially certified as an INGUIDE Level 1 Guideline Panel Member.

Why We Can't Wait for Our Austin Workshop

The U.S. GRADE Network's 18th workshop is just five weeks away! With a focus on assessing certainty of evidence informed by non-randomized studies, this workshop will take place in Austin, Texas - the home of Austin City Limits, the famous Bat Bridge, and some next-level barbecue joints.

We asked some of the workshop facilitators about what they're looking forward to about the trip (besides the chance to talk about GRADE and evidence-based medicine for three days straight!) Here's what they said...

Limited space remains for the workshop. Learn more and register and www.gradeconf.org. 

2022 Recap and What's Next for the U.S. GRADE Network

Thanks to you, 2022 was one of our best years yet.

We were relieved to see you in person, our first time since 2020, at our spring workshop in Chicago last June! Then, we reconvened online for a virtual workshop in November. Overall, we hosted over 100 participants from around the world, stretching from Canada and Spain to Colombia and Sweden. 

Facilitators and participants at the spring 2022 GRADE guideline development workshop in Chicago, IL - the first in-person USGN workshop since March 2020.

We also launched our new two-day virtual Systematic Review workshop offering and welcomed another 48 learners, including a record 13 scholars from across the globe!

We published our takeaways from our early experiences moving our workshop online in response to the Covid-19 pandemic in BMJ Evidence-Based Medicine, which you can read about here.

Finally, we launched the USGN Webinar Series and will host regular free webinars on all things GRADE. For our inaugural session, we were joined by Drs. Gordon Guyatt and Reem Mustafa. You can watch the recording and subscribe to our new YouTube channel here. 

We look forward to seeing you at our 18th Guideline Development Workshop, in person in sunny Austin, Texas, May 17-19, 2023! The workshop will have a special focus on assessing the certainty of evidence informed by non-randomized studies. Registration is now open.