Agent-Assisted Evidence Synthesis: A Survey

Summary

An exploratory systematic map of how AI and LLM agents are being used across the evidence-synthesis pipeline. It reconciles competing stage vocabularies, organizes the retained literature by workflow stage, contribution, evidence type, and setting, and synthesizes what selected studies show about performance measurement, human baselines, disclosure norms, and multi-model design. The retained map is screening- and medicine-heavy, while the reported results are heterogeneous and difficult to compare: metrics, aggregation, reference labels, and deferral can all change the apparent verdict. Guidance repeatedly calls for disclosure of the tool, task, human role, configuration, and verification, but the selected proposals cover different subsets and their evidence notes report no validation or adoption evidence. Ensemble and human–AI deferral designs report conditional gains, while agent reviewer independence remains an open measurement problem rather than an established construct.

Reading list

The works that anchor the map, curated from the selected evidence notes and organized by the survey’s taxonomy, with the method foundations and comparison points that bound its synthesis. Entries link the original paper (DOI where one exists) and our notes on it (library pages where ingested, our evidence notes in the record otherwise).

Method foundations

PaperWhatVenue · YearNotes
Guidelines for Performing Systematic Literature Reviews in Software EngineeringThe SLR method canon for SEEBSE Report · 2007kitchenham2007
Systematic Mapping Studies in Software EngineeringDefines the systematic-map genreEASE · 2008petersen2008
Guidelines for Snowballing in Systematic Literature StudiesSnowballing search guidelinesEASE · 2014wohlin2014
The PRISMA 2020 StatementThe reporting standard for systematic reviewsBMJ · 2021page2021

Landscape and lineage

PaperWhatVenue · YearNotes
Automation of Systematic Literature ReviewsPre-LLM landscape of SLR automation (41 studies, 2006–2020)IST · 2021notes
Automated Support for Searching and Selecting Evidence in Software EngineeringThe medicine-vs-SE adoption gap, measured pre-LLMSEAA · 2021notes
Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-AnalysesStage-by-stage map of where LLMs can assistJMIR · 2024notes
Large Language Models, Updates, and Evaluation of Automation Tools for Systematic ReviewsICASR community report on tools and evaluationSyst Rev · 2024notes
The Phases of Living Evidence Synthesis Using AIPhase model and tool inventory for living evidenceJMIR · 2026notes

Screening

PaperWhatVenue · YearNotes
Enhancing Title and Abstract Screening with GPT-3.5 TurboIn one review case, eight sequential prompt variants produced materially different reported sensitivitiesBMJ EBM · 2023notes
Screening Articles for Systematic Reviews With ChatGPTEarly cross-corpus screening evaluation; stability is not validityJ Comput Lang · 2024notes
GPT-3.5 Turbo and GPT-4 Turbo in Title and Abstract ScreeningSame-prompt comparison: GPT-4 Turbo had higher specificity and similar sensitivityJMIR Med Inform · 2025notes
SESR-Eval: Dataset for Evaluating LLMs in Title-Abstract Screening9-LLM × 24-review benchmark; none of the tested models met the study’s proposed deployment barESEM · 2025notes
Empirical Evaluation of Open-Source LLMs for Paper SelectionAbstract-only evidence: open-weight models screened more conservatively than GPT-4.1SANER-C · 2026notes
LLM-Based Test Oracles: Source-of-Authority TaxonomyA disclosed-adoption exemplar: LLM pre-filter with audits, reported under PRISMAarXiv · 2026notes

Extraction and appraisal

PaperWhatVenue · YearNotes
Agreement in Risk-of-Bias Assessment Between RobotReviewer and Human ReviewersAbstract-only evidence: agreement with the source Cochrane reviews’ own human risk-of-bias judgments ranged from kappa 0.60 (allocation concealment) to 0.04 (blinding of outcome assessors) across domains — a per-domain spread an aggregate accuracy figure would hide; pre-LLM tool, 190 nursing trialsJ Nurs Scholarsh · 2021notes
Accuracy and Efficiency of Machine Learning–Assisted Risk-of-Bias AssessmentsAbstract-only evidence: a rare RCT-grade automation study found noninferiority, not superiorityAnn Intern Med · 2022notes
(Semi)automated Approaches to Data Extraction for Systematic ReviewsLiving review of extraction approaches in social science (23 studies)F1000Research · 2024notes
Data Extraction for Systematic Mapping Study Using a Large Language ModelAbstract-only evidence: an LLM extraction proof-of-concept in SEESEM · 2024notes
Benchmarking Human–AI Collaboration for Common Evidence Appraisal ToolsRetrospective benchmark: agreement-gated deferral outperformed humans alone and LLM ensembles on two easier instrumentsJ Clin Epi · 2024woelfle2024

Ensembles and end-to-end systems

PaperWhatVenue · YearNotes
Ensemble of Deep Learning Language Models for Living Systematic ReviewsFive same-family BERT classifiers: configuration reported, no diversity or correlation statistic measured anywhere; unanimity plus a confidence threshold trades coverage for accuracy — ~98.5% F1 on ~50% of records, against one crowdsourced annotation team’s labelsSyst Rev · 2023notes
Dual-Model LLM Ensemble Reaches Near-Perfect Sensitivity for ScreeningPreprint: cross-vendor OR ensemble reached 99.7% sensitivity on author-adjudicated labels (94.0–94.5% on original labels); no family-diversity ablationmedRxiv · 2025fagerberg2025
LLM-Based Multiagent Collaboration for Abstract ScreeningThe only selected evidence record measuring inter-agent diversity — Spearman 0.48–0.56 among three cross-vendor agents, which a debate round raised while losing the ensemble’s edge; gains are over the best single agent on a benchmark’s original Cochrane inclusion labels, and vendor, scale, and training data vary together, so no factor is isolatedBiol Methods Protoc · 2026notes
A PRISMA-Aligned Agentic Framework for Medical Systematic ReviewsPreprint evaluated on an author-built benchmark: decomposition, structured artifacts, human review, and no redundancymedRxiv · 2026notes

Norms, reporting, and evaluation methodology

PaperWhatVenue · YearNotes
Guidance for Using AI for Title and Abstract ScreeningPre-LLM guidance that rejects fully autonomous thresholding and separately risk-ranks post-truncation optionsBMC Med Res Methodol · 2021notes
Responsible Integration of AI in Rapid ReviewsCochrane Rapid Reviews position statement, endorsing RAISECochrane Ev Synth · 2025notes
Best Practice Methods for Living Evidence Synthesis: An International Modified DelphiPreprint: a 29-expert, three-round Delphi on living-evidence practice; by the authors’ own account consensus was lowest on actionable automation use and highest on general statements, and no voted statement adjudicates AI’s reviewer rolemedRxiv · 2025notes
Leveraging AI for Systematic Reviews: The FRAISR FrameworkPreprint: per-stage machine-readable reporting frameworkOSF · 2024notes
Transparent Reporting of AI in SLRs: PRISMA-trAIceChecklist proposal with human/AI-split flow diagramJMIR AI · 2025notes
HAICO-SLR: Conducting and Reporting Human-AI Collaboration in SLRsPreprint: stage-typed human/AI role guide with dual reporting tablesSSRN · 2026notes
Reproducibility and Data Storage for Active Learning-Aided Systematic ReviewsWhat must be stored, not merely disclosed, for an active-learning screening pass to be reconstructable — a 15-item checklist, never applied to a completed review and demonstrated only against a tool two of its three authors developAppl Sci · 2024notes
LLM4SCREENLIT: Assessing LLM Performance for ScreeningMeasurement reform: why accuracy misleads under class imbalanceIST · 2026madeyski2025

Linked from