AppliedNovember 2026
A fast model's own confidence can decide when to call a bigger one.
- 0.0%
- right when Jev is 95% sure
- 0.0%
- Jev first, same as Opus alone
- 0×
- faster, and 1.7× cheaper
1Abstract
Reasoning models such as Claude Opus answer medical exam questions well, but they think out loud first, which is slow and costs money. System One models answer straight away and report how sure they are. We tested three System One models and four reasoning models on 3,019 orthopaedics questions from MedMCQA. On its own, Jev 1.14 scored 82.8%, against 94.2% for Claude Opus 5.5. But Jev knew when it was right: when it was at least 95% sure, it was right 98.5% of the time, and that happened on about half of all questions. Letting Jev+Claude Opus work together, Jev first and passing only the questions it was unsure about to Opus, scored 94.2%, the same as Opus alone, while answering a typical question 12× faster at 1.7× lower cost. Across the 32 pairings of a fast and a big model that we tried, none scored meaningfully worse than the big model alone.
2Introduction
The most accurate AI models on medical exam questions are reasoning models: ChatGPT, Claude and Gemini write out their reasoning before they answer. That makes them slow and expensive. A typical answer takes 3 to 6 seconds, and every question pays for the reasoning that comes before the answer.
System One models are a new kind of AI that skips that step. Jev from TypeSafe, and Liquid d1-ultra and Liquid d1.1 from Liquid AI, read a question and return an answer straight away, along with how sure they are. TypeSafe released Jev in September 2026. These models are very fast and very cheap, but on their own they are less accurate than the large reasoning models.
That tradeoff only matters if every question has to go to one model or the other. If a fast model's confidence can be trusted, it can answer the questions it is sure about and pass the rest to a reasoning model. The question this work asks is whether a System One model's own confidence is good enough to decide which orthopaedic exam questions need an expensive reasoning model, and which do not.
Orthopaedics gives a clean test: MedMCQA contains 3,019 orthopaedics questions with a known correct answer, enough to measure how often a fast model is right at each level of confidence.
3Methods
Questions. All 3,019 orthopaedics questions in MedMCQA that have a known correct answer. The fast models answered every question. To keep costs low, the reasoning models answered a random set of 500, and every comparison between models uses those same 500 questions.
Models. Three System One models, which answer without reasoning: Jev 1.14 (TypeSafe), Liquid d1-ultra and Liquid d1.1 (Liquid AI). Four reasoning models: DeepSeek v4.1 Flash, GPT-6 Sol (OpenAI), Gemini 3.1 Pro (Google) and Claude Opus 5.5 (Anthropic). DeepSeek was run with reasoning both on and off.
Routing. Jev answers every question first. If its confidence is at or above a cutoff, its answer stands. If not, the question goes to a reasoning model, whose answer is used instead. We tested two cutoffs, 0.95 and 0.80, and chose both before running the reasoning models.
Measures. For each model and each pairing we report accuracy with a 95% confidence interval, cost per 1,000 questions, and the median seconds per question. To check whether a pairing differs from its reasoning model alone, we used an exact McNemar test on the same 500 questions.
A fair test. Every model got the same instructions. Every question and every answer was saved, with code that rebuilds every number in this write-up. Running every test together cost under $9. One fast model, Parallel, was left out because it searches the internet, where these answers can be found.
4Results
Jev knows when it is right
The surer Jev was, the more often it was right.
| How sure Jev was | Share right |
|---|---|
| Under 50% | 50% |
| 50 to 80% | 72% |
| 80 to 95% | 88% |
| 95% or more | 98.5% |
Each model alone
On their own, the reasoning models score higher. Jev is the fastest and among the cheapest.
| Model | Accuracy | 95% range | Cost per 1,000 | Median s |
|---|---|---|---|---|
| Claude Opus 5.5 | 94.2% | 91.8 to 95.9 | $7.25 | 3.88 |
| GPT-6 Sol | 92.8% | 90.2 to 94.8 | $1.32 | 2.73 |
| Gemini 3.1 Pro | 92.2% | 89.5 to 94.2 | $6.47 | 5.93 |
| DeepSeek v4.1 Flash | 91.0% | 88.2 to 93.2 | $0.30 | 1.66 |
| Liquid d1-ultra | 86.8% | 83.6 to 89.5 | $0.005 | 0.38 |
| DeepSeek, no reasoning | 85.8% | 82.5 to 88.6 | $0.059 | 1.04 |
| Liquid d1.1 | 84.6% | 81.2 to 87.5 | $0.002 | 0.35 |
| Jev 1.14 | 82.8% | 79.2 to 85.9 | $0.017 | 0.21 |
The fast models and DeepSeek also answered all 3,019 questions, with nearly the same scores: Jev 83.9%, Liquid d1.1 85.0%, Liquid d1-ultra 86.4% and DeepSeek 90.4%.
Jev first, a reasoning model when Jev is unsure
Every pairing kept its reasoning model's score, including Jev+Claude Opus. Change is in percentage points against the same reasoning model answering every question.
| Reasoning model | Accuracy | Change vs alone | Cheaper | Faster |
|---|---|---|---|---|
| Cutoff 0.9549% sent on to the reasoning model | ||||
| Claude Opus 5.5 | 94.2% | +0.0 | 1.7× | 12× |
| GPT-6 Sol | 93.0% | +0.2 | 1.4× | 8× |
| Gemini 3.1 Pro | 92.2% | +0.0 | 1.6× | 18× |
| DeepSeek v4.1 Flash | 91.0% | +0.0 | 1.1× | 5× |
| Cutoff 0.8032% sent on to the reasoning model | ||||
| Claude Opus 5.5 | 93.6% | −0.6 | 2.5× | 17× |
| GPT-6 Sol | 93.0% | +0.2 | 2.0× | 12× |
| Gemini 3.1 Pro | 91.8% | −0.4 | 2.2× | 26× |
| DeepSeek v4.1 Flash | 90.4% | −0.6 | 1.4× | 7× |
Combining all three fast models instead of Jev alone also matches Claude Opus (94.2%), at 2.5× lower cost.
5Discussion
The result rests on one property: Jev's confidence is calibrated. On the half of questions where it is at least 95% sure, it is right 98.5% of the time, and Claude Opus does no better on those questions. Passing only the other half on loses nothing, which is why every pairing kept its reasoning model's score.
The main gain is speed. A typical answer comes back 5 to 26 times faster, depending on the reasoning model and the cutoff, because half or more of the questions never reach the slow model. The cost savings are smaller, 1.1 to 2.5 times, because the questions that get passed on are the hard ones, which are also the most expensive to answer.
The cutoff is a choice between safety and cost. At 0.95, the Jev+Claude Opus pairing scores exactly what Claude Opus scores alone. At 0.80, a third of questions are passed on instead of half, and the pairing costs 2.5 times less, with a difference from Opus alone that is within chance.
Jev is not a replacement for a reasoning model. On its own it scored 82.8%, far below Claude Opus. Its value is as a fast first filter that knows its limits. Two versions of Jev gave the same results.
What comes next
- Remove the questions that refer to an image the models never saw, and rerun.
- Review the 14 questions every model got wrong: did the models fail, is the answer key wrong, or is the question unclear?
- Report scores by subspecialty: trauma, sports, spine, paediatrics, tumours, hand and joint replacement.
6Limitations
- Source. MedMCQA questions come from Indian medical entrance exams, not US orthopaedic boards or real patient cases.
- Format. Multiple choice and text only. A few questions refer to images the models did not see.
- Sample size. The reasoning models answered a random 500 of the 3,019 questions, so their scores carry wider confidence intervals than the fast models' scores on all 3,019.
- Exams are not care. Doing well on multiple choice questions is not the same as helping patients.
7Availability
The code and every model response are in a private repository while the paper is prepared for submission, along with a summary of every model combination we tried. Email hello@layeredlabs.ai for access.
The questions come from MedMCQA.
Living draft, updated November 2026. This page can change before formal publication.
Jev 1.14