At T-CAIREM 2025 we presented preliminary results from Medevi-RoB, our benchmark for automated Risk of Bias 2 (RoB 2) assessment of randomized controlled trials.
Why automate risk-of-bias assessment?
RoB 2 assesses how study design and reporting affect the trustworthiness of trial results, but it is slow and variable: risk-of-bias analysis takes 6–12 weeks in a typical review, and inter-rater reliability for the overall judgement is only 0.16 even among expert reviewers.
What we did
We evaluated outcome-level RoB 2 judgements from Cochrane reviews in dermatology, neurology and anesthesia, comparing gpt-oss-120b under algorithmic (per-domain) and single-shot chain-of-thought prompting with minimal- and extensive-prompt GPT-4 baselines.
Key findings
- Modern open-source reasoning models with structured prompting substantially outperformed earlier GPT-4 baselines across all RoB 2 domains.
- LLM-derived predictions reached human-level agreement on every domain, with Cohen’s κ from 0.190 (deviations from intended interventions) to 0.573 (randomization process).
- Algorithmic prompting tends to overestimate risk when information is missing; single-shot prompting tends to underestimate it. Overall accuracy is similar (about 72%).
- Cost per paper was $0.0034 for single-shot and $0.0188 for algorithmic prompting.
What’s next
We are completing the evaluation on about 1,000 randomized trials, comparing fine-tuned open models, and extending the framework to ROBINS-I, QUADAS-C and AMSTAR.