Layered Labs

ToolsFebruary 2026

BenchBase: one format and one scoring contract for medical benchmarks

0
medical benchmarks
0
questions in one schema
0
command to score a model

1Summary

BenchBase converts medical multiple-choice benchmarks into one format and scores every model on every benchmark the same way, with one command. It covers five benchmarks and 200,784 questions: MedQA, MedMCQA, PubMedQA, the medical subjects of MMLU, and TRIAGE. It is for researchers who need numbers they can compare across models, papers and time, and it is open source under the MIT license.

2Statement of need

Five benchmarks store the same three things in different places. BenchBase maps each into one schema.

Medical benchmarks ship in different formats. One stores the answer as a letter, another as an index, another as text, and some keep a reading passage next to the question. Before a model can be tested, someone has to write code to read each one.

How a model is asked, and how its reply is read, changes the score. A different prompt, a different temperature, or a looser way of finding the answer in a long reply gives a different number for the same model on the same questions. Most papers choose their own, so two reports of the same model on the same benchmark often cannot be compared.

That makes every new model, and every new benchmark, a small project. BenchBase turns both into a configuration change: one schema for every benchmark, and one fixed way of asking and scoring.

3Design

Every question takes the same path. A row that fails the schema is rejected and never scored.

BenchBase has four parts, and each one makes a single decision.

Converters and the registry. Each benchmark has a converter that maps its rows into the shared schema, and an entry in one registry file with its source and license. Adding a benchmark means writing a converter and adding an entry, and nothing else.

One schema. Every question becomes the same record: the question, an optional reading passage, the options, the answer, its split, and a SHA-256 hash of the question and answer text that stays the same however the options are ordered. The schema decides what is malformed: an empty option, an answer that matches no option, duplicate option keys, or fewer than two options. A row that fails is rejected and logged, never scored.

One scoring contract. Every model gets the same zero-shot prompt, with its version recorded on every result, at temperature 0, and must reply with only the key of the option it chose. The reply must be exactly one of the question's option keys. Nothing is extracted or repaired: anything else counts as no answer and is scored wrong.

A resumable runner. Questions are streamed from Hugging Face one row at a time, so nothing is downloaded up front, and every response is written to disk the moment it returns. An interrupted run loses nothing and continues where it stopped. Comparisons between models use only the questions every model answered, with an exact McNemar test, Holm correction across pairs, and paired bootstrap intervals.

On every merge, GitHub Actions converts each benchmark and publishes it to Hugging Face. GitHub is the source of truth; Hugging Face is the build output.

4Usage

One command runs 100 questions from each of the five benchmarks and writes plain files.

Install from GitHub and run any model on OpenRouter:

git clone https://github.com/Layered-Labs/benchbase-med && cd benchbase-med
pip install -e .
benchbase doctor
benchbase run --model qwen/qwen3-8b --limit 100

--limit 100 runs a random 100 questions from each benchmark; the same --seed gives the same sample. Without it, BenchBase runs every question. Several models at once, --model a,b,c, runs each one and then compares them.

Each run writes plain files that open anywhere: every question and response, metrics per benchmark and split, a one-page report, figures at 300 dpi, and a CSV with one row per scored question.

Anything that speaks the OpenAI chat format can be scored like a model, including a local model, an ensemble, or a retrieval pipeline:

benchbase run --model my-system --base-url http://localhost:8000/v1

Adding a benchmark takes three steps: write a converter, add an entry to the registry, and run benchbase check <key>, which tests the converter against a live sample. Continuous integration runs the same check on every pull request.

5Validation

Source rows converted per benchmark. Grey is what was skipped or rejected.

Every source row is checked against the schema when it is converted. All five benchmarks convert with almost no losses:

BenchmarkSource rowsConvertedRejectedNote
MedMCQA193,155187,0050Test split skipped: its 6,150 answers are not public
MedQA11,45111,4510
MMLU Medical1,2421,242099 items share a hash with another item
PubMedQA1,0001,0000
TRIAGE87861One row has an empty question
Every source row either passes the schema or is rejected and logged. A converter that rejects more than 5% of a split fails.
BenchmarkWhat it isQuestionsLicense
MedMCQAIndian medical entrance exams187,005Apache 2.0
MedQAUS medical licensing exam style, 4 options11,451CC BY 4.0
MMLU MedicalSix medical subjects from MMLU1,242MIT
PubMedQAYes, no or maybe, from research abstracts1,000MIT
TRIAGEMass-casualty triage, 4 options86Pending
All five200,784
The five benchmarks in BenchBase. Every split each source has is converted.

The test suite runs in two parts. The offline tests cover the schema, the converter base class, the registry, and the whole evaluation pipeline against a stub model, so they run anywhere in seconds. The network tests run every registered converter against its live source on Hugging Face. A registry entry without a license fails the tests, so a benchmark that cannot be published cannot be merged.

6Limitations

In MMLU Medical, 99 of 1,242 questions share a hash with another question.
  • Text multiple choice only. Questions with images, and free-text answers, need an image field and their own scoring contract first.
  • Shared hashes. The hash covers the question and answer text only, so short generic questions with the same answer can share one: 99 items in MMLU Medical do. Evaluation adds an item ID that includes the options, so different questions are never merged.
  • Strict scoring. A model that answers correctly but in the wrong format is scored wrong. This is deliberate, but it lowers scores for models that do not follow instructions well.
  • Token budgets. Reasoning models can spend their whole budget thinking and return nothing. Those replies are recorded as truncated and scored wrong; the default budget is 8,192 tokens.
  • Contamination. BenchBase does not check whether a model saw these questions during training.
  • TRIAGE license. TRIAGE is published with its authors' permission while they choose a license.

7Availability

The five converted benchmarks on Hugging Face, with their question counts.

The code is on GitHub at Layered-Labs/benchbase-med under the MIT license. The converted benchmarks are on Hugging Face as Layered-Labs/benchbase-medqa, benchbase-medmcqa, benchbase-pubmedqa, benchbase-mmlu-medical and benchbase-triage.

Each converted benchmark keeps its source's license.

Living draft, updated November 2026. This page can change before formal publication.