Jev benchmarks, evals and calibration

TypeSafe AI

@typesafeai

we announced Jev 1.14 on 10.14 at 1:14pm PT 🔥 a new and improved model, along with experimental features for our dreamers: • images • a new box type • longer context surprise, Jev was already able to see you! 👁️ BUT our bar for production-grade intelligence is quite high, so builders beware, these are preview features. enter the ~*~ danger zone ~*~ at your own risk. 🚨

Shann³

@shannholmberg

how to add a decision model to your AI workflow 1,000 yes/no decisions can cost around: ~$10 on GPT-6 Astra ~$4 on Claude Opus 5.5 ~$0.04 on JEV use language models for planning, writing and coding, and decision models for decisions a decision model can: > check your agent’s output before it ships > approve a draft or request revisions > route a task to the right agent or tool > classify leads, tickets or comments > score fit, quality or priority > check whether content follows your brand rules > pick an option from a shortlist > decide whether to continue, stop or request a review at a much faster speed as well than any regular LLM here’s how to add one 1. find the decisions in your workflow look for repeated steps that need a choice, classification or score start with one decision you can compare against examples you’ve already reviewed 2. define the question and allowed answers give the model the information and references it needs for a content workflow, that could include your draft, voice DNA file, anti slop rules and approved examples ask it to return a choice such as approve, revise or human review 3. connect the answer to an action your software uses that answer to decide what happens next approved drafts move to your review queue, revision requests return to the writing agent, and uncertain cases go to you 4. test before letting it act compare its answers with your reviewed examples if it returns confidence scores, check whether those scores match how often it gets the decision right start with actions you can easily reverse 5. put simple checks first use code for fixed rules, such as required fields and character limits then use the decision model for checks that need context, before starting more expensive work 6. track quality and cost measure cost per completed task, incorrect decisions and how often another model or a person needs to review use the failures to improve your questions, references and allowed answers

OpenRouter

@OpenRouter

1/ Jev, a decision model by @typesafeai, sparked a burst of projects and discussion. We tested it using Ori Eval against popular LLMs on OpenRouter at judging. Jev was >5x faster than the next fastest model, and even its slowest requests beat every other model's median.

idan levin

@0xidanlevin

We just ran Jev on our WebMCP benchmark. The result: basically broke the benchmark. Jev + Mercury 2.5 (a fast, low-cost LLM) using WebMCP solved 100% of the tasks at roughly 112× lower model cost than GPT-6 Astra using computer use with code execution. Compared to Astra using screenshot-based computer use, the model cost was 245× lower (!). We also compared Jev operating the browser with and without WebMCP. We used Browser Use’s open-source Ultrafast, with some improvements to the harness to make it more reliable across the benchmark. Jev’s browser-control accuracy on its own was not amazing - adding WebMCP nearly doubled the number of solved tasks, from 25/49 to 49/49, while reducing model cost by 18% (more on why below). The benchmark and methodology are fully open and reproducible. Full results: https://webmcp.com/benchmark A few words on how the Jev + WebMCP harness works and why this is exciting: Jev receives text as input and a set of discrete options it can choose from. With WebMCP, those options are the tools exposed by the website. At each step, Jev sees the task, the available tools and previous results, then picks what to do next. The limitation is that Jev can’t generate arbitrary text, which you need for tool arguments. For example, it can choose the search_products tool, but it can’t generate the search query itself. So we split the work: Jev picks the tool and Mercury 2.5 generates the arguments if needed. This works well because turns out most of the cognitive load in these tasks is around choosing the right action. The argument generation itself is relatively simple, so we can delegate to a small and very fast model. We used Mercury, which outputs 1,000+ tokens/sec and is very cheap. The result is a pretty simple combination: Jev for tool selection + Mercury for arguments + WebMCP for the interface. It ends up being very reliable, very fast, and very cheap. A few words about Ultrafast and why do we think it underperforms: Without WebMCP, Jev chooses from the page’s controls: which button to click, which field to fill, or which option to select. But choosing a valid button is different from choosing the right next step. The agent still has to navigate menus, understand forms, recover from errors and recognize when the task is actually complete. Our hypothesis is that WebMCP makes the decision space much simpler. Instead of figuring out a sequence of clicks through a website, Jev chooses explicit actions that directly advance the task. @typesafeai itself documents weaker accuracy on questions requiring multiple reasoning steps. WebMCP moves much of that complexity into the website’s tools, leaving Jev with clearer decisions and fewer opportunities to go wrong (in a sense WebMCP "compresses" a sequence of clicks into one tool call). Our modified Ultrafast setup solved 25/49 tasks - that is a result for our particular implementation and benchmark, not a universal limit on Jev or Browser Use. We are open to more harness optimization to get this result to perform better, feel free to directly contribute to the benchmark here: https://github.com/nekuda-ai/WindTunnel Browser-use ultrafast: https://github.com/browser-use/jev-ultrafast

Dan Shipper

@danshipper

we almost never test new foundation models but we've been testing this for ~a week @every and it's pretty wild. the kind of things that will be obviously indispensible in 6-12 months it doesn't produce words as output, it produces probabilities. so it can efficiently act as a judge in cases where you'd need a Fable-level model—but in our testing was 25x faster and 600x lower priced excellent vibe check by @hammer_mt on @every: https://every.to/also-true-for-humans/mini-vibe-check-typesafe-s-jev-judged-everything-i-ve-written-in-0-7-seconds?utm_cta_source=home_main_a_3

nader dabit

@dabit3

Also have been playing with @typesafeai Jev, insane! So many immediate use cases and new apps are possible. What a time to be a builder! Sharing some experiments here starting with: Keystroke oracle / predictive launcher: Your launcher ranks by aliases, fuzzy match, and habit. Jev reads intent: type "the pdf I just downloaded" and the newest PDF is already the top hit with a full confidence on every keystroke, in ~100 ms

ILIAS ISM

@illyism

Now using @typesafeai Jev in http://aiseotracker.com, http://linkdr.com, http://genppt.com, etc AI ends up vibe coding so much AI regex slop if you don't read the code, so I can finally move all this hard-coding to Jev and it's insanely fast! Also for regular LLM calls, it is around 10x faster, 50% cheaper

Wayne Sutton

@waynesutton

Ask Jev anything. Give it a try at askjev.ai It won't answer. It will judge. Let's see if we can get to 1 million questions. @typesafeai 🤝 @convex work great together. @hmartenjoyer @CompleteSkeptic @justKDeng @mikeysee

Ira Bodnar

@irabukht

Jev dropped the price of SEO/GEO fixes by 90% Agents that audit and fix a client's SEO/GEO used to cost us ~$250 Here's where the savings come from: 1/ 30x faster reads of Search Console and PostHog/Mixpanel data 2/ 30x faster checks of what ChatGPT searches on Bing 3/ 30x faster modeling of what users ask Gemini and Claude 4/ 30x faster scans of who ChatGPT and Claude cite 5/ 30x faster analysis of the sources behind those citations 6/ 30x faster gap analysis: why they get cited and we don't 7/ 30x faster fixes across 1,000s of pages on large client sites 8/ 30x faster sorting of which page types ChatGPT cites 9/ 20x faster creation of the pages that make ChatGPT pick you Available in the Ryze AI app and MCP/Claude Connector, link in the 1st comment 👇

jaffa

@dsqjaffa

today i'm releasing Jev for content marketing. still doomscrolling to figure out what to post on social media? that's over now... Jev watches EVERY video in your niche and judges it before it ever reaches you: 1. research: pulls every video in your niche from a database of 12.8M viral videos 2. analyze: Jev watches, studies, and judges each one, the hooks, the formats, the angles, and why they worked 3. create: turns it into a data-backed script, based on proven winners (via Claude) no more guessing on TikTok & Instagram currently available for free in @virlomain + MCP. link below ↓

Florian Darroman

@floriandarroman

Jev is INSANE. I asked 100 Indie Hackers to build a post scheduler with: Jev vs Fable 5.1. The results are unexpected 🤯 (You can clearly see Jev is faster at doing stuff)

Jason Lu

@jasonlu_ai

JEV is changes the world of E2E testing! Same eBay test flow, completed-run medians: Jev: 47s / $0.0067 GPT-5.6 Luna: 62s / $0.0277 Claude Sonnet 5: 79s / $0.4062 Try jev-e2e. github.com/perixtar/jev-e…

Kun Chen

@kunchenguid

almost every day i hear people ask "when should i /compact my session" there's no easy answer because it depends on how likely your future action will need detailed context in the existing window but we have Jev now! introducing compact-adviser - an agent plugin you can use in claude and pi today to help determine whether you're likely at a task boundary that's safe to compact https://github.com/kunchenguid/compact-adviser i built a private eval set from 40 real sessions and manually labeled all the safe vs unsafe checkpoints to evaluate this, and hillclimbed the Jev prompt till it performed quite well i also made it so that the classifier will - optimize for precision (not triggering a compaction prematurely) when context window is small - and gradually shift to optimize for recall (not missing an opportunity to compact) when context window fills up, because the cost of not compacting becomes higher, and at the end the agent will be forced to compact anyway it supports a "hint" mode (just give you a hint and it's up to you to run /compact) vs "auto" mode which runs compaction whenever Jev says it's safe to do so if you have Jev and want to put your compaction on autopilot, try this out and let me know how it goes! support for more harness is coming soon as well