ToolsFebruary 2026
BenchBase: one format and one scoring contract for medical benchmarks
- 0
- medical benchmarks
- 0
- questions in one schema
- 0
- command to score a model
1Summary
BenchBase converts medical multiple-choice benchmarks into one format and scores every model on every benchmark the same way, with one command. It covers five benchmarks and 200,784 questions: MedQA, MedMCQA, PubMedQA, the medical subjects of MMLU, and TRIAGE. It is for researchers who need numbers they can compare across models, papers and time, and it is open source under the MIT license.
2Statement of need
Medical benchmarks ship in different formats. One stores the answer as a letter, another as an index, another as text, and some keep a reading passage next to the question. Before a model can be tested, someone has to write code to read each one.
How a model is asked, and how its reply is read, changes the score. A different prompt, a different temperature, or a looser way of finding the answer in a long reply gives a different number for the same model on the same questions. Most papers choose their own, so two reports of the same model on the same benchmark often cannot be compared.
That makes every new model, and every new benchmark, a small project. BenchBase turns both into a configuration change: one schema for every benchmark, and one fixed way of asking and scoring.
3Design
BenchBase has four parts, and each one makes a single decision.
Converters and the registry. Each benchmark has a converter that maps its rows into the shared schema, and an entry in one registry file with its source and license. Adding a benchmark means writing a converter and adding an entry, and nothing else.
One schema. Every question becomes the same record: the question, an optional reading passage, the options, the answer, its split, and a SHA-256 hash of the question and answer text that stays the same however the options are ordered. The schema decides what is malformed: an empty option, an answer that matches no option, duplicate option keys, or fewer than two options. A row that fails is rejected and logged, never scored.
One scoring contract. Every model gets the same zero-shot prompt, with its version recorded on every result, at temperature 0, and must reply with only the key of the option it chose. The reply must be exactly one of the question's option keys. Nothing is extracted or repaired: anything else counts as no answer and is scored wrong.
A resumable runner. Questions are streamed from Hugging Face one row at a time, so nothing is downloaded up front, and every response is written to disk the moment it returns. An interrupted run loses nothing and continues where it stopped. Comparisons between models use only the questions every model answered, with an exact McNemar test, Holm correction across pairs, and paired bootstrap intervals.
On every merge, GitHub Actions converts each benchmark and publishes it to Hugging Face. GitHub is the source of truth; Hugging Face is the build output.
4Usage
Install from GitHub and run any model on OpenRouter:
git clone https://github.com/Layered-Labs/benchbase-med && cd benchbase-med
pip install -e .
benchbase doctor
benchbase run --model qwen/qwen3-8b --limit 100
--limit 100 runs a random 100 questions from each benchmark; the same --seed gives the same sample. Without it, BenchBase runs every question. Several models at once, --model a,b,c, runs each one and then compares them.
Each run writes plain files that open anywhere: every question and response, metrics per benchmark and split, a one-page report, figures at 300 dpi, and a CSV with one row per scored question.
Anything that speaks the OpenAI chat format can be scored like a model, including a local model, an ensemble, or a retrieval pipeline:
benchbase run --model my-system --base-url http://localhost:8000/v1
Adding a benchmark takes three steps: write a converter, add an entry to the registry, and run benchbase check <key>, which tests the converter against a live sample. Continuous integration runs the same check on every pull request.
5Validation
Every source row is checked against the schema when it is converted. All five benchmarks convert with almost no losses:
| Benchmark | Source rows | Converted | Rejected | Note |
|---|---|---|---|---|
| MedMCQA | 193,155 | 187,005 | 0 | Test split skipped: its 6,150 answers are not public |
| MedQA | 11,451 | 11,451 | 0 | |
| MMLU Medical | 1,242 | 1,242 | 0 | 99 items share a hash with another item |
| PubMedQA | 1,000 | 1,000 | 0 | |
| TRIAGE | 87 | 86 | 1 | One row has an empty question |
| Benchmark | What it is | Questions | License |
|---|---|---|---|
| MedMCQA | Indian medical entrance exams | 187,005 | Apache 2.0 |
| MedQA | US medical licensing exam style, 4 options | 11,451 | CC BY 4.0 |
| MMLU Medical | Six medical subjects from MMLU | 1,242 | MIT |
| PubMedQA | Yes, no or maybe, from research abstracts | 1,000 | MIT |
| TRIAGE | Mass-casualty triage, 4 options | 86 | Pending |
| All five | 200,784 |
The test suite runs in two parts. The offline tests cover the schema, the converter base class, the registry, and the whole evaluation pipeline against a stub model, so they run anywhere in seconds. The network tests run every registered converter against its live source on Hugging Face. A registry entry without a license fails the tests, so a benchmark that cannot be published cannot be merged.
6Limitations
- Text multiple choice only. Questions with images, and free-text answers, need an image field and their own scoring contract first.
- Shared hashes. The hash covers the question and answer text only, so short generic questions with the same answer can share one: 99 items in MMLU Medical do. Evaluation adds an item ID that includes the options, so different questions are never merged.
- Strict scoring. A model that answers correctly but in the wrong format is scored wrong. This is deliberate, but it lowers scores for models that do not follow instructions well.
- Token budgets. Reasoning models can spend their whole budget thinking and return nothing. Those replies are recorded as truncated and scored wrong; the default budget is 8,192 tokens.
- Contamination. BenchBase does not check whether a model saw these questions during training.
- TRIAGE license. TRIAGE is published with its authors' permission while they choose a license.
7Availability
The code is on GitHub at Layered-Labs/benchbase-med under the MIT license. The converted benchmarks are on Hugging Face as Layered-Labs/benchbase-medqa, benchbase-medmcqa, benchbase-pubmedqa, benchbase-mmlu-medical and benchbase-triage.
Each converted benchmark keeps its source's license.
Living draft, updated November 2026. This page can change before formal publication.