Systematic Reviews and Meta Analysis

Each risk of bias tool has benefits and drawbacks to use. Here are some considerations for conducting a risk of bias assessment:

• Typically a Risk of Bias tool is designed to assess bias for a single type of study (such as a Randomized Controlled Trial)
• Some Risk of Bias tools can be adapted to other types of studies, such as ROBINS-I, which is used to assess Non-randomized studies of intervention (NSRI)
• For observational or non-experimental studies, it may be necessary to use an additional tool to assess risk of bias, such as the JBI critical appraisal tools, which features targeted RoB assessments for a variety of study types.
• Risk of Bias assessment should be completed by at least two separate people and reported as a table (AHRQ)
• For Risk of Bias of RCTs and NSRIs, prior to undertaking the RoB assessment, the team will need to select which specific results from the included trials to assess (Cochrane)
  • Trials usually contribute multiple results to a systematic review, several risk-of-bias assessments may be needed for each trial, although it is unlikely to be feasible to assess every result for every trial in the review (Cochrane)
  • An approach that focuses on the main outcomes of the review (the results contributing to the review’s ‘Summary of findings’ table) may be the most appropriate approach (Cochrane)
• Each article should be assessed based on the domains of the RoB tool, then the team should make a judgement about the overall risk-of-bias and categorize thusly:
  • Low risk-of-bias - The trial is judged to be at low risk of bias for all domains for this result.
  • Some Concerns - The trial is judged to raise some concerns in at least one domain for this result, but not to be at high risk of bias for any domain.
  • High Risk of Bias - The trial is judged to be at high risk of bias in at least one domain for this result.
    OR
    The trial is judged to have some concerns for multiple domains in a way that substantially lowers confidence in the result. (cochrane)
• If your review features multiple study types:
  • Use 1 tool for all studies:  Offers direct comparison and consistent data. However, rigidity of tools may overlook study-specific domains.
  • Use different tools for each type of study:  Each tool has a benefit to use and will contain domains specific to the type of study. However, the data won't provide easy comparisons due to inconsistencies.

• It is inadvisable remove articles from inclusion due to the results from a Risk of Bias assessment, as that will introduce bias into your review and skew results.

According to PRISMA 2020, [Item 11] the manuscript of the review should clearly name the name of the tool and version used. If modifications were made, it should be indicated as well. Report the methods and steps used to assess bias in the body of the manuscript.

The Risk of Bias itself should be reported in a table. Most frequently, reviews use a red/green/yellow "traffic light" approach to coding bias. While others may incorporate symbols (*, +, -, ?) either separately or in addition to the traffic light coding to indicate concerns about levels of bias. Other reports may use numerical values to indicate levels of bias, but these have a tendency to be more vague about the results of the RoB assessment.

Additionally, including study data from the review authors in the appendix adds further transparency to the process.