Critical appraisal isn't fault-findingIt's deciding how much to trust a paper's results

First, a common misconception to clear up.

Critical appraisal is not about grading a paper pass or fail. No study is perfect, so counting flaws would condemn every paper — and kill the discussion.

The real questions are these: which parts of the results can be trusted, and which deserve reservation? And how far do they apply to your own patients, practice, or research?

Pointing out a flaw is a starting point, not a conclusion. Keep that in mind and the quality of your appraisal changes.

There are seven places to look when calibrating that distance.

First, the PICO or research question: what is the study actually asking? Second, the design: does it fit the question? Third, how participants were selected: any selection bias? Are baselines balanced across groups?

Fourth, the primary outcome: is the yardstick for "effective" appropriate? Fifth, the size and precision of the effect: how big, and how certain? Sixth, confounding, bias, and missing data: were the things that distort results dealt with?

Seventh and last, applicability: does it transfer to the patients, setting, and questions you actually work with? Let's take these in the order they tend to come up at journal club.

PICO. P is Patient/Population — who was studied: age, sex, diagnosis, severity, comorbidities. I is Intervention/Exposure — the treatment, test, or care being evaluated: a new drug, a rehabilitation protocol, surgery.

C is Comparison — what the intervention is measured against: standard care, placebo, or observation without intervention. O is Outcome — what changed as a result: survival, symptom improvement, complication rates, quality of life.

In short: for whom, doing what, compared with what, measured how. Work these four until you can state them in a single sentence. Skip this, and every later judgment goes blurry.

Study design. RCT, cohort, or case-control? What matters is not "RCTs are simply superior" but whether the design fits the question being asked. RCTs are poor at catching rare adverse events, and some questions can't ethically be randomized at all.

Participant selection. Look at the inclusion and exclusion criteria, and at Table 1. Overly strict exclusions narrow who the results apply to. And if baselines differ between groups, you can't tell whether later differences come from the intervention or from where the groups started.

The primary outcome. Two checks: did the pre-specified primary outcome show a significant difference, or is the paper leading with a "win" on secondary outcomes only? And is that outcome an indicator that actually matters to patients and to people in the field?

Effect size and precision. Here's an invitation to move beyond the p-value alone: look at the effect measure — risk difference, risk ratio, NNT — and at the width of the confidence interval.

"Statistically significant" and "clinically meaningful" are different things. With a large enough sample, even a tiny difference can produce a small p-value. In your presentation, discuss not just whether a difference exists but how that difference should be read.

Confounding, bias, missing data. For observational studies: how were confounders adjusted? What proportion dropped out, and how were they handled? Was there blinding? Funding sources and conflicts of interest get checked here too.

Applicability. The last point — and the one that should generate the most discussion. Where do the study's participants and setting resemble your own, and where do they differ?

A journal club that gets past "there is evidence" to "how — or whether — we'd use this in our own workplace" is doing exactly what a journal club is for.

How far can AI help with these seven checks?

AI is good at the prep work. It can extract the PICO, lay out the design and primary outcome, and list the limitations the authors acknowledge. Up to that point, it's an extension of summarizing, and genuinely useful.

Beyond that is not AI's job. Calibrating the distance can only be done by someone who has seen the actual numbers and figures. Is the baseline imbalance big enough to overturn the result? How should this confidence interval be read clinically? Should it apply to your patients?

The AI summary is the entry point; the original is the evidence. Nowhere does that principle matter more than in critical appraisal.

Critical appraisal means taking notes while cycling through seven perspectives. Scatter those notes across chat logs and loose paper, and they're hard to find before the presentation — and harder still months later when you wonder, "How did I rate that paper?"

Paperfy keeps the paper's PDF, AI summary, and figures on one page, and you can overwrite the AI summary with your own appraisal as you go.

Corrections like "the AI summarized it this way, but the original defines the primary outcome differently" become a lasting record of your read on the paper. Radio scripts are editable the same way, so you can regenerate the audio and review your conclusions by ear before the meeting.

Critical appraisal isn't passing judgment on a paper — it's translating its findings into terms that matter for your own work.

Use the seven checkpoints to calibrate distance, let AI handle the prep, and make the final calls from the numbers in the original. Master this, and journal club shifts from a room where people get grilled to a room where discussion actually happens.

Start with just one of the seven: state the PICO in a single sentence, and feel the difference it makes.