PaperfyPAPERFYHome
Start freeLog in
Paperfy Guides
risk of bias assessmentSystematic review6 minFor anyone doing critical appraisal or review work

Risk of Bias Assessment:Ask About Distortion, Not Quality

Ever been asked to “assess the risk of bias in this paper” and, not really knowing what to look at, vaguely written “probably low risk”? Start with the terminology. Risk of bias assessment isn't scoring the paper. It's assessing how much room the study's methods leave for its results to deviate from the truth. Once that clicks, you'll know exactly where to look.

Published: 2026-07-22

Turn a paper PDF into a 3-minute radio show

No registration required. Try a paper PDF you want to read.

Try without signing up

In this article

Clean results and bias-resistant methods are different thingsThe lens changes with the study designWhat to look at in an RCTWhere to read: Methods, registration records, supplementsWhy an AI summary can't judge risk of biasRecord each domain as a judgment plus one line of justificationListen first, then assess in the PDFFinally: reasonable assessors disagreeStart with Paperfy

Clean results and bias-resistant methods are different things

A common trap is the intuition that a study with statistically significant, tidy results must be low risk. If anything, it's the reverse: biased methods are perfectly capable of producing tidy results.

That's why risk-of-bias assessment looks at the methods before the results. The basis for the judgment is the description of the methods, not the shine of the findings.

The lens changes with the study design

A basic premise: what counts as bias risk depends on the design. RCTs, observational studies, and diagnostic accuracy studies each have their own standard assessment tools.

Established tools exist for each of those designs. Which one you use depends on your review type, your team, and your target journal — check before you start. Below, we'll use an RCT to illustrate the shared logic.

What to look at in an RCT

For RCTs, the standard checks start with the allocation method (is the randomization sequence generation described?) and allocation concealment (was there a mechanism preventing anyone from predicting upcoming assignments?).

Then blinding (were participants, those delivering the intervention, and above all the outcome assessors unaware of group assignment?), missing data (how many dropped out, and how were they handled?), and selective reporting — do the pre-registered outcomes match what the paper reports?

Where to read: Methods, registration records, supplements

This information is scattered — mainly across the Methods section, but also in trial registration records (traceable by registration number), protocol papers, and supplementary materials.

“Not reported” is itself a finding. If allocation concealment is never mentioned, that absence feeds into the assessment. The ground rule is: never read silence as “it was probably fine.”

Why an AI summary can't judge risk of bias

By now the reason is visible. A summary is a compression of what is written — and the two keys to this assessment, what isn't written and what conflicts with the registration record, can't survive into a summary even in principle.

That doesn't make AI useless here. Asking it to “pull every passage about allocation, blinding, and missing data from this paper” is genuinely helpful preparation. But the final judgment — including whether that extraction was complete — belongs to someone who has read the original.

Record each domain as a judgment plus one line of justification

For each domain, record not just the judgment but the reasoning behind it: “Blinding: some concerns — no mention of outcome assessor blinding (Methods p.5).”

Records with the evidence attached feed straight into team consensus discussions and peer review responses. One sentence of reasoning outlasts the verdict itself.

Listen first, then assess in the PDF

A tool like Paperfy can help you get oriented: drop the PDF and listen to the paper as a roughly 3-minute radio show covering background, methods, key findings, and limitations. You can also read the script.

But the actual assessment — checking what isn't reported, cross-checking registration records, judging each domain — happens in your own PDF reader. Paperfy is the entry point; reading the original PDF for the details is your job.

Finally: reasonable assessors disagree

Risk-of-bias assessment is a task where even trained assessors diverge. Don't treat any single person's judgment as final; plan for uncertain items to be discussed by the team.

The goal of this process isn't to pronounce with confidence — it's to be able to argue from evidence.

Key takeaways

  • The assessment was based on the Methods, not the impression left by the results.
  • The team confirmed an assessment tool suited to the study design.
  • “Not reported” was never read as “probably fine.”
  • Every judgment was recorded with one line of reasoning.

Turn a paper PDF into a 3-minute radio show

No registration required. Try a paper PDF you want to read.

Try without signing up

Systematic review

Explore This Topic

Read the full cluster for this workflow, from first steps to practical use.

Browse all guides

Read Next

These guides continue the same workflow.

systematic review vs meta-analysisSystematic Review vs. Meta-Analysis: One Is a Process, the Other Is a Toolhow to do a scoping reviewHow to Do a Scoping Review: Draw the Map, Don't Chase the Answersystematic review methodologySystematic review methodology: what separates it from a very thorough literature review