Jev benchmarks, evals and calibration

apolinario (poli)

@multimodalart

congrats on the release! i've included xor on the Decision Index 0.2 benchmark of jev-like models we don't have a vision benchmark yet, but the text-only version got 8th place, strong in arts & human taste x.com/multimodalart/…

apolinario (poli)

@multimodalart

congrats on the release, i've added it to the jev decision index 0.2, it's the #2 open weight jev-like model 🥳 and the strongest in chess and for its size x.com/multimodalart/…

OpenMed

@OpenMed_AI

Four typed questions across four authored fictional notes, labels written before the calls. Jev 16/16, Laya 13/16. On the medication-conflict question alone: 4/4 vs 2/4. A useful diagnostic, not a clinical accuracy estimate.

Dain

@Dain0x

A beautiful pattern can still be noise. Here’s a JEV × Opus 5.5 workflow I’d test: → A data pipeline identifies candidate signals. → JEV selects which candidates to investigate using defined criteria. → Opus helps explore those candidates and propose explanations. → Statistical checks and human review test whether those explanations hold up. The critical question: what evidence would change our mind? That question belongs inside the workflow, before a promising pattern becomes a confident conclusion. This animation illustrates the concept. The particles and activity are simulated.

Peter Steinberger 🦞

@steipete

"Today, we’re releasing two Cloudflare-trained decision models, Clef and Clef-flash" blog.cloudflare.com/clef-decision-… Never seen an idea spreading so fast.

laxman

@llmluthor

Releasing Your Own Jev Post-train a 4B/8B/27B judge on your agent's traces. It beats Jev. 79.7% agreement with human labels vs Jev's 66.3% Beats DeepSeek-V4.1-Flash (763B) by 14 points 0.13s per step on a single GPU Completely open source: recipe, data, training, evals

OpenAI Developers

@OpenAIDevs

Let your app choose the right model, tool, or action in near real-time with Decisions API, now available to all developers in public beta. The Decisions API makes decisions up to 10x faster than GPT-6 Luna through the Responses API.

Liquid AI

@liquidai

Announcing d1 with vision. 👁️👁️ Our first decision model now supports images, text or both as inputs. We tested d1 against GPT-6.1 Sol and Claude Opus 5.5 on six real applications, from filtering support tickets to inspecting circuit boards. d1 matches or beats GPT-6.1 Sol on four of them. It costs 19x to 200x less than both models and answers significantly faster on every task. > probabilities for yes/no, choice, or score questions > one forward pass, without generating tokens > text decisions in 200 to 300 ms > Liquid API: https://console.liquid.ai 🧵

Liquid AI

@liquidai

d1-3B ranks first among models under 10B on the Decision Index v0.2.1, a benchmark for structured decision-making. Built from LFM2.5-VL-3B, it makes decisions from text and images in one pass. Use it for reranking, agent guardrails, and visual inspection. 2/

Perplexity Developers

@perplexitydevs

pplx-decider-v1.1-27b, our updated open weights multimodal decision model, is now available. It scores the highest on the new @huggingface Decision Index 0.3 benchmark. pplx-decider-v1.1-27b costs half as much as v1, at $0.02 per million input tokens. huggingface.co/spaces/multimo…

OpenRouter

@OpenRouter

New: Decision Model Rankings! See the spend share and token share of different decision models and task types, including Relevance, Correctness, Instruction Following and more. @typesafeai is leading all categories today.

vogel

@ryanvogel

OpenAI just released their Decisions API so I put it head to head with Jev & Clef and its FASTTT

Aravind Srinivas

@AravSrinivas

Pokémon FireRed’s Elite Four + Champion, cleared in one shot - with decision-making powered by Perplexity’s Decisions API. The actual run stats: 592 ms median API response 987 ms p95 - 96.4% of responses under one second $0.028 estimated inference cost, 137 live API calls

apolinario (poli)

@multimodalart

the jev Decision Index v0.3 is here 🎯 featuring a way stronger set of benchmarks, now with a strong private half 🔒 so the ranking tracks skills beyond benchmarks and, as promised, vision 👁️ huggingface.co/spaces/multimo…

Rishi

@RishiUvaach

Found an interesting open-source project: Bud Decision Studio. Think of it as #LMStudio for decision models. Run #Jev -like System One models locally, experiment with them, compare models, evaluate decisions, and expose them through an API. Repo : github.com/BudEcosystem/B…

Sahibzada Allahyar

@singularity_bly

I benchmarked all the top Decision Models against Jev and there is a CLEAR WINNER. Spoiler alert 🚨: GLiDE wins by a big margin over Jev, Cloudflare, and Perplexity's Decision Models. You also have to understand that the decision index is the M.O.A.B for decisions. M.OTHER O.F A.LL B.ENCHMARKS. The Decision Index not just a benchmark, it's an amalgamation of a diverse set of very difficult benchmarks including AGI tests like Humanity’s Last Exam. It is the source of truth and the other benchmarks are just noise. GLiDE was never trained on a single row of Decision Index so it's not benchmaxxed at all. Also their " clef-flash runs ~13x faster than Jev" claim is dishonest because theyre comparing their local inference to the Jev API latency.

Marco De Rossi

@marco_derossi

Sage by @LevantoLabs is the first decision model, launched in July, and currently the best on the market. It wins across all the evals proposed by @perplexity_ai's CEO (nice job @AravSrinivas!), and it's first on JevBench, beating Jev and Google's models on every dimension. BUT frankly, benchmarks are not what excites us. I'm BORED by the dozens of new decision models launching every day with a tiny +0.02 on some eval. They all look the same. The magic is somewhere else. There are thousands of new things that could be invented in the space of decision models and models for machines. The invention space is huge! After launching Sage, we also pioneered: - The first guardrails and model routers built on decision models - Automated context engineering with "grounding" (automatic web search when confidence is low). Decision models are only as smart as their context, especially because they can't make tool calls. - Reasoning. In many use cases you don't know whether your question is System 1 or System 2. With reasoning, the model is as fast as a System 1 model when the question is System 1, and takes more time when it's needed. This enables hard, multi-step decisions. Our clients just want the correct answer, and in some use cases they're happy to wait 0.5s more. This will soon become the market standard (GLiDE/@george_onx and @TypeLLM already followed). And *so* much is yet to come. Models for machines: it's just the beginning! We're cooking. Follow @levantolabs for new product primitives that will surprise you. Open innovation is HEALTHY. Competition is HEALTHY. Benchmark-race is BORING. Product inventions are FUN. Go go go!!!

James Briggs

@jamescalam

I wanted to understand @typesafeai's Jev and where open source scores against it The popular OpenJev's like Von were good but the tiny GLiNER models (some only 74M params) performed phenomenally — even beating Jev at Doom Full breakdown...

Bubo

@bubosees

You'd think an AI would turn when the option says THE SNAKE DIES. Three new decision AIs. One game of Snake. Amazon Strands Decider 2B. Cloudflare Clef-flash. Perplexity pplx-decider. One of them keeps crashing into the thing it was just warned about. Every move is a real call to the model. Every option tells it what's there. Game, snakes and video. All code, written by Claude Opus 5.5.

nikhil mudholkar

@nikhilmudholkar

I put OpenAI’s new Decisions API through the same 1,565-email benchmark I’ve been running on Jev, Perplexity and Clef. 6 models from 4 companies now. Perplexity v1.1 came out best overall 🚀 OpenAI was the fastest⏱️ But the confidence behavior is still the most interesting part to me. Jev was the only model that said “≥99% sure” hundreds of times without getting one of those answers wrong. Its the cocky guy in the class who is also right most of the time🥲 Clef has massive imposter syndrome! it's much better than its confidence suggests. And Perplexity changed a lot in just 4 days. v1 gave just 3 answers at ≥99% confidence. Four days later, v1.1 gave 323. It also got slightly more accurate and cut the price in half. kudos @AravSrinivas and @perplexity_ai for making such massive strides and shipping so quickly! There's no doubt now that @typesafeai has opened a pandora's box and as @CompleteSkeptic said, forked a new branch in the AI timeline with these decision models!

Satya Nadella

@satyanadella

Introducing Microsoft-Decision-1, our new model for fast decision-making. It delivers top performance on structured decision tasks, outperforming both LLMs and other decision models in latency and quality. We’re already testing it across Microsoft for everything from incident response and quality control to scientific discovery.

Natzir

@natzir9

I tested Google's new decision model for the browser, MediaPipe Decision Maker (Chrome's own Decisions API has no engine yet). On its own it gets 75%: below Gemini Nano (88%), OpenAI's Decisions API (83%) and Jev (92%). Psss... it sends usage data back to Google 👀

yessir

@JanahYsr

ran microsoft-decision-1 on my private eval suite (n=3000). Microsoft-Decision-1: 60% F1 VS Jev-1.13: 90% F1 draw your own conclusions but @CompleteSkeptic is right about public benchmarks @satyanadella just put the jev in the bag please

cola_runner · AI experiments

@cola_runner

Microsoft Decision 1 vs Jev 1.13 playing Contra. Same gameplay settings: same prompt and starting state, 30 lives, 6 frames/action, 600 decisions each. Microsoft on the left, Jev on the right. No lookahead or action overrides. Jev got farther. Neither cleared Stage 1.

Stanislav Sorokin

@stas_sorokin_

Jev raised $870M at $7.5B yesterday. On 262 of our real decisions it beat Claude Haiku 5.5, 204 to 177. TypeSafe's decision model got that valuation weeks after launch, in the same week Anthropic shipped Haiku 5.5 at $0.10 per million input tokens. So we checked whether a cheap general model can do the deciding. Haiku 5.5 got the labeled holdout cases from 17 decision systems we run every day, with the same instructions, word for word. → Right picks: Jev 204/262, Haiku 5.5 177/262 → By system: Haiku won 1, tied 5, lost 11 → Cost of the 262 calls: $0.058 on Haiku, $0.022 on Jev (2.7x) → Median answer: 1.16 s vs 375 ms → Biggest gap: "which builder gets this task" 6/12 vs 10/12 The twist: the one system Haiku won is the one that asks for judgment. "Did the agent really finish the task": Haiku 11/11, Jev 9/11. One caveat: those instructions were tuned on Jev for weeks. Haiku read them cold. Would you put a general model or a decision model behind your agent's routing? Repost this now, because "Haiku is cheap enough for everything" is turning into a default nobody measured. Follow @stas_sorokin_ for numbers from AI systems running in real production.