Risk of Bias Assessment:Ask About Distortion, Not Quality

A common trap is the intuition that a study with statistically significant, tidy results must be low risk. If anything, it's the reverse: biased methods are perfectly capable of producing tidy results.

That's why risk-of-bias assessment looks at the methods before the results. The basis for the judgment is the description of the methods, not the shine of the findings.

A basic premise: what counts as bias risk depends on the design. RCTs, observational studies, and diagnostic accuracy studies each have their own standard assessment tools.

Established tools exist for each of those designs. Which one you use depends on your review type, your team, and your target journal — check before you start. Below, we'll use an RCT to illustrate the shared logic.

For RCTs, the standard checks start with the allocation method (is the randomization sequence generation described?) and allocation concealment (was there a mechanism preventing anyone from predicting upcoming assignments?).

Then blinding (were participants, those delivering the intervention, and above all the outcome assessors unaware of group assignment?), missing data (how many dropped out, and how were they handled?), and selective reporting — do the pre-registered outcomes match what the paper reports?

This information is scattered — mainly across the Methods section, but also in trial registration records (traceable by registration number), protocol papers, and supplementary materials.

"Not reported" is itself a finding. If allocation concealment is never mentioned, that absence feeds into the assessment. The ground rule is: never read silence as "it was probably fine."

By now the reason is visible. A summary is a compression of what is written — and the two keys to this assessment, what isn't written and what conflicts with the registration record, can't survive into a summary even in principle.

That doesn't make AI useless here. Asking it to "pull every passage about allocation, blinding, and missing data from this paper" is genuinely helpful preparation. But the final judgment — including whether that extraction was complete — belongs to someone who has read the original.

For each domain, record not just the judgment but the reasoning behind it: "Blinding: some concerns — no mention of outcome assessor blinding (Methods p.5)."

Records with the evidence attached feed straight into team consensus discussions and peer review responses. One sentence of reasoning outlasts the verdict itself.

In Paperfy, the PDF under assessment and its AI summary and figures share one page, which shortens the loop of find the passage → judge → record.

Add each domain's judgment and rationale to the summary, and a record of the whole risk-of-bias assessment stays in your library with the paper — easy to revisit when you move on to overall certainty of evidence, as with GRADE.

Risk-of-bias assessment is a task where even trained assessors diverge. Don't treat any single person's judgment as final; plan for uncertain items to be discussed by the team.

The goal of this process isn't to pronounce with confidence — it's to be able to argue from evidence.