Research

Benchmarks for AI‑assisted critical appraisal.

Every claim Medevi makes about model performance is measured against expert human reviewers, on public datasets, with confidence intervals. These are our current results.

Benchmark 01 · Preliminary · T-CAIREM 2025

Medevi-RoB: LLM judgements match expert agreement on every RoB 2 domain.

273 outcome-level RoB 2 judgements (154 RCTs, 34 systematic reviews), no fine-tuning. Ground truth is the consensus of two or more independent Cochrane reviewers.

Pre-registered protocol · OSF, Sept 2025 →

Cohen's κ by domain, 95% CI · hover for detailMedevi (best approach)Human reviewers (Minozzi et al.)

0.00.20.40.60.8

Agreement with expert consensus — Cohen's κ (95% CI)
Domain	Best approach	Medevi κ	Human IRR (Minozzi et al.)
D1 · Randomization process	Single-shot CoT	0.573 (0.454–0.678) Moderate	0.45 (0.37–0.53) Moderate
D2 · Deviations from intended interventions	Single-shot CoT	0.190 (0.040–0.342) Slight	0.04 (−0.06–0.14) Slight
D3 · Missing outcome data	Single-shot CoT	0.341 (0.205–0.469) Fair	0.22 (0.14–0.30) Fair
D4 · Measurement of the outcome	Algorithmic	0.525 (0.388–0.656) Moderate	0.27 (0.19–0.35) Fair
D5 · Selection of the reported result	Algorithmic	0.373 (0.267–0.475) Fair	0.30 (0.22–0.38) Fair
Overall risk of bias	Per-domain algorithmic	0.213 (0.125–0.297) Fair	0.16 (0.08–0.24) Fair

Dataset

Four Cochrane reviews, 714 RCTs: Eczema (291), Chronic plaque psoriasis (204), Parkinson's disease (154), Neuraxial anesthesia (65).

Cost per paper · gpt-oss-120b

$0.0034 single-shot chain-of-thought; $0.0188 algorithmic per-domain.

Error profile

Algorithmic prompting tends to overestimate risk (19.3%); single-shot tends to underestimate it (18.3%). Accuracy is similar (~72%).

Benchmark 02 · ICAM 2026

MERSQI & NOS-E: parity on one scale, a clear gap on the other.

139 peer-reviewed medical education studies from five systematic reviews, scored with Cook & Reed (2015) operational definitions. Bootstrap 95% CIs.

Read the ICAM 2026 summary →

Total-score ICC (ICC2) by model and prompting strategy · hover for detailZero-shotPer-domainHuman–human

MERSQI · human ICC 0.79

GPT-5-nano

GPT-5-mini

GPT-5.2

NOS-E · human ICC 0.82

GPT-5-nano

GPT-5-mini

GPT-5.2

Human extraction vs. Medevi
Metric	Human reviewers	Medevi
Time per paper	65–130 min	29–57 s
Cost per paper	$21.67–$43.33 at $20/h	$0.10 zero-shot · $0.57 per-domain
MERSQI agreement	ICC 0.79	ICC 0.755 [0.704, 0.801] · substantial
NOS-E agreement	ICC 0.82	ICC 0.345 [0.268, 0.424] · fair

What's next

From experimenting with AI to relying on it, with evidence.

Extend the benchmark to ROBINS-I, QUADAS-2, PROBAST and AMSTAR.
Complete Medevi-RoB on ~1,000 randomized trials across specialties.
Evaluate title/abstract and full-text screening for medical education reviews.
Compare fine-tuned open models and hybrid NLP to cut cost further.