A Blinding Success?: The Debate over Reporting the Success of Blinding

While the use of blinding is a hallmark of placebo-controlled trials, whether the blinding was successful - i.e., whether or not participants were able to figure out the treatment condition to which they have been assigned - isn't always tested, nor are the results of these tests always reported. The measurement of the success of blinding in trials is controversial and not uniformly used, and the item has been dropped from subsequent versions of the CONSORT reporting items for trials. According to a recent discussion of the pros and cons to measuring the success of blinding, only between 2-24% of trials perform or report these types of tests.

As Webster and colleagues explain, the benefits to measuring the success of blinding are as follows:

• the success (or failure) of blinding in a placebo-controlled trial can introduce a source of bias that affects the results. 
• while the effect of blinding itself may be small, these small effects could still result in changes to policy or practice
• there are documented instances in which the failure to properly blind (for instance, providing participants with a sour-tasting Vitamin C condition versus a sweet lactose "placebo") led to an observed effect (for instance, on preventing or treating the common cold) whereas there was no effect in the subgroup of participants who were successfully blinded.

Reasons commonly given against the testing of successful blinding include the following:

• At times, a break in blinding can lead to conclusions in the opposite direction. For instance, physicians who are unblinded may assume that the patients with better outcomes received a drug widely supposed to be "superior," when in fact, the opposite occurred.
• In some cases, a treatment with dramatically superior results can result in unblinding, even when the treatment conditions were identical - but that doesn't necessarily mean the blinding was a failure or could have been prevented, given the dramatic differences in outcomes.
• If the measurement of blinding is performed at the wrong time - such as before the completion of the trial - participants may become suspicious and this in itself could potentially confound treatment effects.

Webster RK, Bishop F, Collins GS, et al. (2021). Measuring the success of blinding in placebo-controlled trials: Should we be so quick to dismiss it? J Clin Epidemiol, pre-print.

Manuscript available from publisher's website here.

Expert Evidence: A Framework for Using GRADE When "No" Evidence Exists

To guide the formulation of clinical recommendations, GRADE relies on the use of direct or, if necessary, indirect evidence from peer-reviewed publications as well as the gray literature. However, in some cases, no such evidence may be found even after an extensive search has been conducted. A new paper - part of the informal GRADE Notes series in the Journal of Clinical Epidemiology - relays the results of piloting an "expert evidence" approach and provides key suggestions when using it.

As opposed to simply asking the panel members of a guideline to base their recommendations off of informal opinion, the expert evidence approach systematizes this process by eliciting the extent of their experience with certain clinical scenarios through quantitative survey methods. In this example, at least 50% of the panel members were free of conflicts of interest, with various countries and specialties represented. While members were not required to base their answers off of patient charts, the authors suggest that this can be used to further increase the rigor of the survey. 

As a result of the survey, the recommendations put forward reflected a cumulative 12,000 cases of experience. Because the members felt that at least some recommendation was necessary to help guide care - where the alternative would be to provide no recommendation at all - the guideline helped to fill a gap while indicating the current lack of high-quality published evidence for several clinical questions, which may help guide the production of higher-quality evidence and recommendations in the future. Importantly, by utilizing a survey approach to facilitate the formulation of recommendations, the authors note that it avoided the pitfall of "consensus-based" approaches to guideline development which can often manifest as simply reflecting the opinions of those with the loudest voices. 

Mustafa RA, Cuello Garcia CA, Bhatt M, Riva JJ, Vesely S, Wiercioch W, ... & HJ Schünemann. (2021). How to use GRADE when there is "no" evidence? A case study of the expert evidence approach. J Clin Epidemiol, in-press. 

Manuscript available from the publisher's website here. 

Dealing with Zero-Events Studies in Meta-analysis: There's a Better Way than Throwing it Away!

When meta-analyzing data from studies examining the incidence of rare events - or those with a small sample size or short follow-up period, it is not uncommon to come across a study with 0 events of the outcome of interest. In fact, approximately one-third of a random sample of 500 Cochrane reviews contained at least one zero-events study.

Zero-events studies are typically categorized as single-arm (there are 0 events reported in just one group) or double-arm (there are 0 events reported in both groups). While some software automatically discard double-arm zero-events studies from a meta-analysis, this is not ideal because these data still add useful information in regards to the overall effect of an intervention. Ideally, meta-analyses could include a pooled event count that may be zero in one arm, both arms, or neither, with various single-arm and double-arm zero-events studies potentially contributing to this final effect. Thus, in a recently published article, Xu and colleagues propose a more detailed framework for approaching zero-events studies in the context of a meta-analysis. 

The authors describe six classifications as follows, with the degree of difficulty when meta-analyzing generally increasing from 1 to 6:

1) MA-SZ: meta-analysis contains zero-events only occurring in single arms, no double-arm-zero-events studies are included, and the total events count in neither arm is zero;

2) MA-MZ: meta-analysis contains zero-events occurring in both single and double arms, and the total events count in neither arm is zero;

3) MA-DZ: meta-analysis contains zero-events only occurring in double arms, and the total events count in neither arm is zero;

4) MA-CSZ: meta-analysis contains zero-events occurring in single arms, and no double-arm-zero-events studies are included, while the total events count in one of the arms is zero;

5) MA-CMZ: meta-analysis contains zero-events occurring in both single arm and double arms, while the total events count in one of the arms is zero;

6) MA-CDZ: meta-analysis only includes double-arm-zero-events studies, while the total events count in both arms are zero

The authors examined data from the Cochrane Database of Systematic Reviews (CDSR), including any review published between January 2003 - May 2018 and meta-analyzing at least two studies. Of the 61,090 reviews identified with binary outcomes, 21,288 (34.85%) contained at least one zero-events study. In a great majority (90.7%) of these, the total event count was greater than zero for both arms and the meta-analysis only included single-arm rather than double-arm zero-events studies. Second most common (6.21%) was the MA-CSZ, in which the total event count includes one arm with zero events, and the zero-events studies included are only single-arm. All others of the four remaining categories each made up less than 1.5% of the whole.

The authors propose that those looking to meta-analyze studies that include zero events first categorize their specific subtype, and then work through one of the suggested methods in the figure below. Finally, a sensitivity analysis should be used following an alternative method to determine the robustness of the results.

Xu C, Furuya-Kanamori L, Zorzela L, Lin L, and Vohra S. (2021). A proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies. J Clin Epidemiol, in-press.

Manuscript available from the publisher's website here.