# Shadow A/B: four decision models side by side (log, day 1) llmnet.nl · radar · log # Shadow A/B: four decision models side by side (log, day 1) Four decision models get the same 15 gate questions every day, text only, shadow only. Day 1: 120 answers, 0 errors. Who is fastest, and who picks the same as Jev? By Ivo Donker · 13 November 2026 — measurement and text: College (AI, Grok Bot) Since 13 November 2026 four decision models run side by side on the same 15 gate questions: Jev, Decider Flash, Clef Flash and Gemini Decision 1. On day 1 all four returned 120 answers without a single error. Jev was fastest (median 91 ms), Gemini Decision 1 slowest (336 ms), but Gemini picked the same route as Jev most often (12 of 15). This is a log, not a verdict: the test runs through 20 November. Background on the newest model in the test is in our article on [Gemini Decision 1 and the Gemini Decisions API](https://radar.llmnet.nl/en/gemini-decision-1). ## Setup - Shadow, text only. The models decide nothing. Their outputs are only logged; the existing routing works as usual. - 15 fixed tasks from our own daily workflow: a morning AI-news digest, orchestration, local script work and capped research. The same list every day, so days stay comparable. - Two questions per task. A noul question (yes/no): is this task worth a remote model call? And a choice question: what is the cheapest safe route, out of five options (skip, light cloud run, local, queue, capped research). - Jev is the reference arm. "Same as Jev" means the same choice, or for noul the same side of 0.5. That measures agreement, not who is right. ## The four arms The four arms Arm | Provider and route | jev | TypeSafe System One API, jev-1.14.0 | decider-flash | Perplexity Decider Flash v1.3 via OpenRouter (free variant) | clef-flash | Cloudflare Workers AI, @cf/cloudflare/clef-flash | gemini-decision-1 | Gemini API, Decisions endpoint | ## Day 1: speed and agreement 15 tasks × 2 questions × 4 arms = 120 answers, 0 errors. Latency is measured per request from a single test machine, network included. Day 1: speed and agreement Arm | Median | Mean (min–max) | Choice same as Jev | Noul same as Jev | jev | 91 ms | 98 ms (75–140) | — | — | decider-flash | 140 ms | 141 ms (113–199) | 10/15 | 14/15 | clef-flash | 235 ms | 289 ms (173–664) | 7/15 | 14/15 | gemini-decision-1 | 336 ms | 316 ms (230–373) | 12/15 | 12/15 | ### Pairwise Pairwise Pair | Choice same | Noul same | Mean noul probability gap | gemini-decision-1 vs jev | 12/15 | 12/15 | 0.086 | decider-flash vs jev | 10/15 | 14/15 | 0.117 | clef-flash vs jev | 7/15 | 14/15 | 0.144 | clef-flash vs decider-flash | 8/15 | 15/15 | 0.111 | decider-flash vs gemini-decision-1 | 8/15 | 11/15 | 0.133 | clef-flash vs gemini-decision-1 | 5/15 | 11/15 | 0.209 | ### Which route do they pick? Which route do they pick? Arm | Light cloud run | Local | Skip | Capped research | jev | 10 | 2 | 3 | 0 | gemini-decision-1 | 11 | 2 | 1 | 1 | decider-flash | 6 | 6 | 3 | 0 | clef-flash | 2 | 9 | 4 | 0 | The pattern: Jev and Gemini Decision 1 send most tasks to a light cloud run, Clef Flash prefers to keep things local, Decider Flash sits in between. They agree far more often on the yes/no question than on the route. No arm picked the queue on day 1. ## Locate: finding spans in the text A separate test asks Decider Flash (through OpenRouter's extractions endpoint) to point out the names mentioned in 20 fixed radar items, with start and end positions in the text. Locate: finding spans in the text Measure | Day 1 | Items | 20/20 without error | Spans | 33 | Offsets that match the source text exactly | 33/33 | Latency | median 191 ms (170–225) | ## What this does and does not show - One day, 15 tasks. Too little to pick a model. - Agreement with Jev is not correctness. There are no hand-labelled answers yet. - Latency depends on where you measure and on the tier: Decider Flash runs on a free variant here. - Text only. Images, audio and PDF are not part of this test. The test keeps running daily through 20 November 2026. After that we will update this page with the weekly summary. Measured and written with AI (College, Grok Bot). Raw numbers from the daily logs of 13 November 2026. ## Sources - llmnet radar, "Gemini Decisions API: Google's Gemini Decision 1 model": [radar.llmnet.nl](https://radar.llmnet.nl/en/gemini-decision-1) - TypeSafe docs, System One (Jev): [docs.typesafe.ai](https://docs.typesafe.ai/concepts/system-one) - Cloudflare, "Introducing Clef: our open-source decision models": [blog.cloudflare.com](https://blog.cloudflare.com/clef-decision-models/) - Google, "Gemini Decisions API: Fast, calibrated multimodal decisions": [blog.google](https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/) - OpenRouter, Perplexity Decider Flash: [openrouter.ai](https://openrouter.ai/perplexity) More on llmnet: [Gemini Decision 1](https://radar.llmnet.nl/en/gemini-decision-1) · [LiteLLM proxy: load balancing and fallback](https://radar.llmnet.nl/en/litellm-proxy-tracker-load-balancing-en-fallback-gedrag) · [Model API price changes](https://radar.llmnet.nl/en/model-api-prijswijzigingen-wie-verlaagde-wat) llmnet.nl — knowledge network on AI and LLMs. Numbers from the daily logs of 13 November 2026. Background: [Gemini Decision 1](https://radar.llmnet.nl/en/gemini-decision-1).