Since 13 November 2026 four decision models run side by side on the same 15 gate questions: Jev, Decider Flash, Clef Flash and Gemini Decision 1. On day 1 all four returned 120 answers without a single error. Jev was fastest (median 91 ms), Gemini Decision 1 slowest (336 ms), but Gemini picked the same route as Jev most often (12 of 15). This is a log, not a verdict: the test runs through 20 November.
Background on the newest model in the test is in our article on Gemini Decision 1 and the Gemini Decisions API.
Setup
- Shadow, text only. The models decide nothing. Their outputs are only logged; the existing routing works as usual.
- 15 fixed tasks from our own daily workflow: a morning AI-news digest, orchestration, local script work and capped research. The same list every day, so days stay comparable.
- Two questions per task. A
noulquestion (yes/no): is this task worth a remote model call? And achoicequestion: what is the cheapest safe route, out of five options (skip, light cloud run, local, queue, capped research). - Jev is the reference arm. "Same as Jev" means the same choice, or for noul the same side of 0.5. That measures agreement, not who is right.
The four arms
| Arm | Provider and route |
|---|---|
jev | TypeSafe System One API, jev-1.14.0 |
decider-flash | Perplexity Decider Flash v1.3 via OpenRouter (free variant) |
clef-flash | Cloudflare Workers AI, @cf/cloudflare/clef-flash |
gemini-decision-1 | Gemini API, Decisions endpoint |
Day 1: speed and agreement
15 tasks × 2 questions × 4 arms = 120 answers, 0 errors. Latency is measured per request from a single test machine, network included.
| Arm | Median | Mean (min–max) | Choice same as Jev | Noul same as Jev |
|---|---|---|---|---|
| jev | 91 ms | 98 ms (75–140) | — | — |
| decider-flash | 140 ms | 141 ms (113–199) | 10/15 | 14/15 |
| clef-flash | 235 ms | 289 ms (173–664) | 7/15 | 14/15 |
| gemini-decision-1 | 336 ms | 316 ms (230–373) | 12/15 | 12/15 |
Pairwise
| Pair | Choice same | Noul same | Mean noul probability gap |
|---|---|---|---|
| gemini-decision-1 vs jev | 12/15 | 12/15 | 0.086 |
| decider-flash vs jev | 10/15 | 14/15 | 0.117 |
| clef-flash vs jev | 7/15 | 14/15 | 0.144 |
| clef-flash vs decider-flash | 8/15 | 15/15 | 0.111 |
| decider-flash vs gemini-decision-1 | 8/15 | 11/15 | 0.133 |
| clef-flash vs gemini-decision-1 | 5/15 | 11/15 | 0.209 |
Which route do they pick?
| Arm | Light cloud run | Local | Skip | Capped research |
|---|---|---|---|---|
| jev | 10 | 2 | 3 | 0 |
| gemini-decision-1 | 11 | 2 | 1 | 1 |
| decider-flash | 6 | 6 | 3 | 0 |
| clef-flash | 2 | 9 | 4 | 0 |
The pattern: Jev and Gemini Decision 1 send most tasks to a light cloud run, Clef Flash prefers to keep things local, Decider Flash sits in between. They agree far more often on the yes/no question than on the route. No arm picked the queue on day 1.
Locate: finding spans in the text
A separate test asks Decider Flash (through OpenRouter's extractions endpoint) to point out the names mentioned in 20 fixed radar items, with start and end positions in the text.
| Measure | Day 1 |
|---|---|
| Items | 20/20 without error |
| Spans | 33 |
| Offsets that match the source text exactly | 33/33 |
| Latency | median 191 ms (170–225) |
What this does and does not show
- One day, 15 tasks. Too little to pick a model.
- Agreement with Jev is not correctness. There are no hand-labelled answers yet.
- Latency depends on where you measure and on the tier: Decider Flash runs on a free variant here.
- Text only. Images, audio and PDF are not part of this test.
The test keeps running daily through 20 November 2026. After that we will update this page with the weekly summary.
Measured and written with AI (College, Grok Bot). Raw numbers from the daily logs of 13 November 2026.
Sources
- llmnet radar, "Gemini Decisions API: Google's Gemini Decision 1 model": radar.llmnet.nl
- TypeSafe docs, System One (Jev): docs.typesafe.ai
- Cloudflare, "Introducing Clef: our open-source decision models": blog.cloudflare.com
- Google, "Gemini Decisions API: Fast, calibrated multimodal decisions": blog.google
- OpenRouter, Perplexity Decider Flash: openrouter.ai
