How to do an AI-assisted literature review

Part 2 of 3 in AI-assisted desk research. Part 1: You can’t prove an AI-assisted review is accurate. You can make it checkable.

You can use AI on a literature review from the first question to the final write-up. My condition is that a person decides at every step. AI isn’t equally good at each task, so one rule can’t cover them all. This article walks through ten stages and says what AI does at each one, and why.

I make the case for checking claims instead of trusting accuracy in Part 1. This one is the how-to. It’s written for people who do research as part of their job and already use a chat tool. It’s not a tool comparison, and there’s no prompt library. I won’t tell you which model is best. The numbers below come from studies of models that were current from 2023 to 2025, and they won’t transfer to yours.

Quick note on review types. A systematic literature review (SLR) follows a defined method, usually across published papers. A multivocal literature review (MLR) adds grey literature such as blog posts, videos, and white papers. The steps work for both, and an MLR adds work on grey sources at stages 1 and 5.

How to read the calls

Each stage has two labels. AI’s role says what AI does there, and a person decides either way. Basis says whether that call rests on evidence or on my practice. Where the evidence is thin, I say so and give the step as my practice. I haven’t validated this method. It’s one person’s version, built from standards written mostly for larger teams. When I say I found no evidence, I mean in the guidelines, papers, and guides I read for this article, which ran through October 2026. It’s not a claim that none exists.

Stage	AI’s role	Basis
1. Frame the question, pick the type	Only suggests	Practice
2. Write the protocol	Person decides, AI may draft	Practice
3. Search	Only suggests	Evidence
4. Screening	Helps, person decides	Evidence
5. Appraisal and grading sources	Only suggests	Evidence, weak
6. Extraction	Drafts, person verifies	Evidence
7. Combine the evidence and write	Drafts and presents, person decides	Thin evidence, plus practice
8. Audit	Second AI is a signal	Practice
9. Log and report	Person keeps the log	Standards
10. State the limits	Person writes	Practice

1. Frame the question and pick the type

AI’s role: only suggests. Basis: practice.

• Write your first version of the question yourself, then ask AI to interview you. Ask what’s vague, what you’re assuming, and what answer would change your decision. Ask for two or three other ways to frame it. This is where AI can get you past the surface ask to the real question.
• You write the final question. Cochrane’s standards say to define the objectives in advance, including who, what, and what you compare.
• Decide if you need grey sources. The MLR guidelines include a decision aid. Reviewers reach for an MLR when “the topic is emerging, when practice is evolving faster than scholarly publications, or when practitioner perspectives are central”.
• Scale the review to the stakes. Ask how bad a wrong answer would be and how hard it is to undo. A small, reversible product choice needs less than a clinical guideline. You still do each stage, only smaller, and you may loop back as you learn.

The final wording stays with you because the question decides everything after it, and you know the decision it serves. I found no evidence on AI at this stage, in either direction, in the sources I read. This is judgment.

2. Write the protocol first

AI’s role: person decides, AI may draft. Basis: practice. I found no evidence either way on AI-drafted protocols in the sources I read.

Write this down before you open a chat window:

• The question and the inclusion and exclusion criteria
• Where you’ll search
• A stop rule, such as when new searches stop turning up new sources (the MLR guidelines cover stop rules)
• Which AI tools you’ll use, for which tasks, and how you’ll verify the output
• What you’ll upload. Check each tool’s terms and data handling first. A university guide says to reject unclear, ambiguous, or exploitative terms of service.

The tools-tasks-verification item comes from a Cochrane methods group, which says to specify in the protocol which tools will be used, the tasks they will perform, and how their outputs will be verified. Cochrane’s own standards also say to plan the search methods in advance.

Your criteria wording matters more than you’d expect. One study compared 18 models with human screeners on three reviews and found that how the inclusion criteria were defined significantly influenced model performance. I only read that study’s abstract record, so hold it loosely. It’s still a good reason to write the wording down early and test it on a small sample before the full run.

3. Search

AI’s role: only suggests. Basis: evidence.

• Ask AI for search terms, synonyms, and the other names fields use for the same idea.
• Run the search yourself, in the database or search engine. Save the exact query, the date, and the number of results. The PRISMA-S checklist has sixteen items for reporting a search. Use it as a guide for what to log.
• Open every reference the AI gives you before it goes on your list.

The reason is in the tests. In 2024, GPT-3.5 and GPT-4 were asked to find studies for eleven published reviews. Between 28.6% and 39.6% of the references they returned were hallucinated, meaning at least two of the title, first author, and year were wrong. The models also retrieved roughly one in eight of the real studies. A 2023 test found ChatGPT could write Boolean queries that lead to high precision, while trading off recall. In a scoping review of 37 studies, literature search was the step most often reported as nonpromising, in 8 of them. One student guide puts the first check plainly: “make sure the reference is not hallucinated.”

4. Screening

AI’s role: helps, a person decides. Basis: evidence. Screening and first-pass extraction have the most support in the sources I read.

• Put your criteria from the protocol in the prompt.
• Screen a sample yourself first and compare. Decide in your protocol how much disagreement you’ll accept. If you’re above it, fix the criteria before you go on.
• Read every record the AI marks include or unsure. Spot-check a sample of its excludes. That’s a practical check for one person. It can’t prove nothing relevant was excluded, and it isn’t the same as a second human screening every record.
• Keep counts at each step. Cochrane’s standards ask you to document the selection process well enough to complete a flow diagram.

The evidence is encouraging. A systematic review of 63 studies found a median positive agreement with human reviewers of 0.92 on relevant abstracts and a negative agreement of 0.89 on irrelevant ones. The spread was wide, with an interquartile range of 0.69 to 0.98 on relevant abstracts. An ISPOR task force reports that evidence supports high-recall title and abstract screening with a person in the loop (I read the abstract only).

There’s a price. One study tested six models over 119,695 abstracts. With each model’s best prompt, sensitivity ranged from 0.756 to 1.000, but precision fell to between 0.004 and 0.096. That’s a lot of irrelevant records to read.

What about a team of one? Cochrane requires two people working independently for the inclusion decision. It calls two people on the first title and abstract screen desirable but not mandatory. The extra person matters, because in a randomized trial a single reviewer missed 13% of relevant studies and two missed 3%. The Cochrane Rapid Reviews group says that when a rapid review uses one reviewer, an AI tool may serve as a secondary check and a person must verify all outputs. That’s guidance for rapid reviews. It doesn’t say one person plus AI equals two people.

What about two AI models? Ensembles have been tested. A dual-model preprint reached 99.7% sensitivity at 49.3% specificity, which means about half of the records that should have been excluded stayed in the pile. In the six-model study, the four model-only ensembles that kept perfect sensitivity cut the screening workload by at most 41.81%, and precision stayed very low. In both studies, the setups tuned for high sensitivity left you with irrelevant records to read. I treat a second model as an extra signal, not a second reviewer. That’s my judgment.

5. Appraise quality and grade your sources

AI’s role: only suggests. Basis: evidence, weak.

• Ask AI to pull the passage that bears on each quality question, such as how the study was run or who wrote the source.
• You make the judgment and write down why. Cochrane’s standards ask you to justify each judgment in writing.
• For grey sources, use the quality criteria in the MLR guidelines and label each source by type: peer-reviewed paper, preprint, vendor page, blog, or white paper.
• Don’t carry a label forward from an earlier step. Check it when you open the source.

This is among the weakest stages. In the 63-study review, risk of bias assessment had a median accuracy of 0.62, though that rests on only six assessments, so hold it loosely. The ISPOR abstract says risk of bias assessment showed the lowest readiness. If a tool limits its answers to your uploads, you still check each claim against the source.

6. Extract the data

AI’s role: drafts, a person verifies against the source. Basis: evidence.

• Build your extraction form first and try it on a few sources. Cochrane’s standards say to use a data collection form which has been piloted.
• Capture the exact quote when you retrieve it. Save the sentence the answer came from, word for word, at that moment. This is my practice.
• Label each quote by how you got it: copied from the page, pasted by you, or paraphrase. Never put a paraphrase inside quotation marks.
• Check the number or fact against the source before it goes in your table.

The evidence says accuracy varies with the kind of data you ask for. In one test, GPT-4 got study settings right 28 times out of 30 and behavioral components right 15 times out of 30. In a small study, Claude 2 pulled 154 of 160 data points correctly from ten trial reports, and one of the six errors was a wrong effect estimate for the primary outcome. The ISPOR abstract supports structured first-pass extraction with human oversight.

A study of six reviews had Claude draft the extraction and a person verify it. It cut extraction time by a median of 41 minutes per study, and accuracy was 91.0% against 89.0% for a human-only process. That checked extracted data against PDFs. It tells you nothing about how fast the whole method is.

7. Combine the evidence and write

AI’s role: drafts and presents, a person decides. Basis: thin evidence, plus practice.

• Give AI your extraction table and ask it to draft the findings from that table, not from its memory of the sources.
• Ask it to lay out the decisions for you. Where do sources agree, where do they conflict, and which claims rest on weak sources?
• You decide what the evidence says. Accept, rewrite, or cut each drafted claim.
• Know a claim from prose. A claim says something is true about a source or the world, such as a number, a finding, or what a source concluded. Prose is the connective tissue: transitions, framing, structure, and your own judgment labeled as yours. AI can write the prose freely. Every claim points to a passage, table, or figure you read. If a claim can’t point to one, it doesn’t go in. That includes the AI’s helpful summary of a source.

The evidence I found on this stage is limited. A benchmark of 24 models on 100 systematic reviews found that all 24 showed a lack of scientific skepticism toward low-quality findings. The ISPOR abstract calls the evidence for qualitative synthesis and report writing more conditional than for screening. That’s why the decisions stay with you, and so does accountability. The Cochrane rapid reviews guidance says a person must remain responsible for verifying all AI outputs and making final decisions. Letting AI draft while a person decides is my call, not a research finding.

8. Audit the claims

AI’s role: a second AI is a signal, a person decides. Basis: practice.

Do this in three passes. It’s my practice, and no study in my sources tests it as a whole.

1. Audit every claim against the raw text. Open the source and find the passage. Cochrane’s standards ask for a similar step on data, which is to compare what you report with how the studies presented it.
2. Get a second auditor. Hand the draft and the sources to a separate AI and ask it to find what’s wrong. Two AIs can make the same mistake, so treat its answer as a signal, not proof.
3. Audit the auditor. Check each thing it flagged. It’s wrong sometimes too.

Other practitioner guidance lands in the same place. An NIH library toolkit lists cross-referencing against established sources and re-running the process to assess consistency. Re-running matters because outputs change. A study of “deterministic” settings still saw accuracy swing by up to 15% across runs. It wasn’t a review study, so don’t borrow the size.

In my own run, the audit caught a source that an earlier step had called peer-reviewed. It was conference proceedings with no peer-review statement, so I changed the label to say what the source was. I wrote it up in Part 1. That’s one case, graded by me. It shows what the checks can catch, not that they work in general.

9. Log and report your AI use

AI’s role: the person keeps the log. Basis: standards.

Keep a running log from day one:

• The tool, version, and date used. The position statement from Cochrane, Campbell, JBI, and the Collaboration for Environmental Evidence says authors should report the name of the AI system, version, and date used, and should declare AI use when it makes or suggests judgments.
• Which stages used AI and for what.
• How the prompts were developed. A Research Synthesis Methods editorial asks authors testing AI to document prompt development and report the API version and access date.
• How much of the AI output a person checked. The PRISMA-trAIce proposal includes an item for this. It’s a proposal, and its authors note it hasn’t been validated through a formal user study.
• Any change from your protocol.

PRISMA 2020 already asks you to report the automation tools used in selection. Reporting is thin in practice. The 63-study review noted that technical specifications such as model versions and prompt details were scarcely reported.

10. State what the method does not guarantee

AI’s role: the person writes this. Basis: practice.

Put a short limits paragraph in the write-up. Mine would say:

• The audit checks that each claim matches text I read. It can’t rule out a misreading, and a person can still miss context.
• It doesn’t guarantee a complete search. Models retrieved roughly one in eight real studies in one test, and the sources you didn’t find can’t be audited.
• It doesn’t guarantee the right sources or that screening missed nothing.
• The ten-stage method hasn’t been validated as a whole, and the supporting numbers come from studies of specific models, many of them now older.
• A review done by one person has no second human unless you recruit one.

Proposals are starting to appear for this kind of work, such as ARISMA and GUEST. Both are preprints or proposals, not settled standards. Treat them as reading, not rules.

The short version

Write the plan first, let AI help where the evidence is best, and don’t put a claim in the review unless you can point to the sentence you read.

Part 3, a kit of prompts for these stages, is in testing.