Using AI for Paper Screening:Let It Set the Order, Not Make the Call

Having AI score each paper's relevance and sorting the pile from most to least likely is the safest, most effective use of AI in first-pass screening.

If AI only sorts, the judge doesn't change. Only the order of the work changes — every paper is still guaranteed a human look.

With the most relevant papers clustered at the top, your "include" decisions come early and your feel for the criteria stabilizes fast.

The tail end becomes a long run of exclusions, but it goes quickly because your judging rhythm is already set. The mental load is nothing like grinding through a randomly shuffled mix.

So can you skip the bottom 30% the AI rated least relevant? As a rule, no.

AI scores are swayed by how abstracts happen to be written, and some papers that meet your criteria will always land low in the ranking. Excluding papers without looking at them bakes systematic misses into the review. In a review that claims comprehensiveness, the principle is that every record gets a human look.

Once AI joins the process, your records should reflect it. Keep separate columns for the AI's relevance score and for the human's final decision and reasoning.

That way you can later measure how often the AI diverged from human judgment — and honestly write in your methods that AI was used for prioritization while all decisions were made by humans.

In reviews requiring independent dual screening or agreement-rate checks, AI use must fit within the team's established procedures.

Confirm with your research team, your supervisor, and your target journal whether — and how — AI can appear in your process and your write-up. Consensus in this area is still forming, and going it alone carries risk.

Papers that survive screening move on to full-text review. From here the work is paper-by-paper, and the tools change with it.

This is where Paperfy earns its keep. Upload each candidate PDF and you get one page per paper with the AI summary, figures, tables, and original text. Add your eligibility findings and reasons to the summary and save them there. Use other tools for first-pass speed, and this layer for second-pass rigor.

Let's be honest: even sorted by AI, 1,800 records are still 1,800 records. There's no magic here.

Even so, a better order and systematic record-keeping genuinely reduce the exhaustion. The real battleground for time is the seconds you spend hesitating on each record. Fix your criteria, hold anything uncertain, and keep a steady pace.