Jev benchmarks, evals and calibration

Anand Prasad

@theanandprasad

you can turn any low-cost open-source LLM into a Jev like fast decision model without training it. Here's how to do it: An LLM doesn't write text directly. At every step it scores every word in its vocabulary for how likely that word is to come next, and then it picks one. Those scores are called logits. Suppose you want to know whether a customer email is angry. The usual way is to ask the model, wait while it writes a paragraph, and then try to pull a yes or no out of the response. The trick is to never let it write anything. Give it the email content and the question, and end your prompt with "Answer:". Your full prompt will look something like this: [ Message: "Third time I've contacted you. Still broken." Is this customer angry? Answer yes or no only. Answer: ] The model reads the prompt once and scores every possible next word. You look at two of those scores. "yes" scores 8.1 "no" scores 5.9 Convert those two scores into probabilities and you get 90% yes and 10% no. (Use softmax function -- google it) Your code then uses that number directly. If it's above 0.8, escalate the ticket. Congrats, you have created your own Jev! Here’s why it works: 1. It's fast. Reading a prompt happens in one parallel pass. Writing is the slow part because the model writes one word at a time. Here the model never writes. 2. It's cheap. Output tokens are the expensive ones, and this produces none. 3. There's nothing to parse. You always get a fixed output, and never a paragraph. 4. You get a confidence level along with the answer. A 0.55 means the case should go to a human or a bigger model. A 0.98 means you can act on it. I tried running the code on llama 3.3 70b (via openrouter) and the accuracy was 21/22 (95%). Cost per query: $0.000013, latency: ~500ms

大伟｜AI × Web3

@david1989_zhu

这条新闻最值得警惕的，不是“24小时覆盖Vercel约13%的付费团队”，而是把覆盖率直接当成产品价值。按报道，Jev不写文本，只做判断；这类产品切中的，是开发流程里高频、重复的决策节点。 我没参与Jev落地，下面是基于团队交付经验的判断。当时看到这个数字，我原来会先问：模型到底有多强？后来逐渐发现，团队持续付费更看重一件事：它能否稳定减少人工审核和返工，而不是演示时答得多漂亮。 现在我会先定清决策边界、误判后的回退方式和单次调用成本，再看它是否进入关键链路。“覆盖”来得快是好信号，但留得住、算得过账，才是商业闭环。

amVT

@cruzex100

This is pretty good.. My smoke test earlier: Accuracy: Jev 0.727 Soft accuracy: Jev 0.580 · Laya 0.471 Calibration (ECE, lower is better): Jev 0.144 · Laya 0.213 Latency: Jev ~710ms · Laya ~30-40ms Cost per decision: Jev ~$0.0004 (API) · Laya ~$0 (self-hosted)

Ramarao Kadiyala

@broadrangeAI

Reviewing Jev: fast, structured decisions for AI workflows. Vendor figures: 70–500 ms, $0.042/M input tokens, $0 output. My takeaway: pair low-cost decisions with clear controls and measurable outcomes. Full post: linkedin.com/posts/activity… #AIEngineering #BroadrangeAI

Zhilin Wang

@zhilinjerrywag

#Jev made one thing very clear: Not every AI model needs to generate. Some models should just make really good decisions. So we’re cooking Mev at @metix_ai , a recruiting-native decision model built around job-candidate matching, ranking, and hiring signals. Jev → Mev. More soon.

nocoo

@zhengli

Jev 学费还是要付的，刚开始觉得，每次请求这么便宜，每个月 $5 妥妥够用，一个真实任务下来就教做人：好好研究官方 skill，学会如何正确使用 state 和按分组问问题。

Min(building Moss, hiring now)

@zeroxmin

Great article from Hrishi! So despite all the hype, JEV is a great observer for agent runs (progress, time/cost-to-finish, trace labeling), but not a security gate, and not yet a laziness detector. Still sounds good to plan JEV into our roadmap and explore some use cases for onchain agents!

yuri

@yurinakanishi33

昨日のJevハッカソンで発表した「Jevを使った俳句生成」について訂正があります。 発表では「JevのChoiceで、ひらがなを1文字ずつ生成している」と説明しましたが、間違いでした。 実際には、季語・名詞・助詞・動詞・副詞・結びの定型句などの単語をコード側であらかじめ用意し、それらを組み合わせて5音・7音の完成行候補まで先に作っていました。 Choiceに渡していたのは、その時点の文字列から、いずれかの完成行に到達できる「次の1文字」だけです。 さらに、候補が1つならコードで確定。複数ならChoiceが返した確率の上位3候補を再正規化し、コード側でもう一度抽選していました。そのため、Choiceが選んだ文字を必ず使っていたわけでもありません。 つまり、単語や文章の道筋をコード側で先に用意したうえで、Choiceが返した確率を使ってコード側で分岐する実装でした。申し訳ありません。 添付動画は、コード側で用意していた語彙と単語のつながりに関する制約を外した版です。残した制約は、5・7・5の区切りと、使用できる文字を1字1音として扱える73字に限定する文字種制限だけです。 毎回、ここまでの句、季節・状況・気分などの文脈、残りの音数、73字すべてをChoiceに渡しています。Choiceが返した全73候補の確率をコード側で再正規化し、その確率を重みに次の1字を抽選。これを17回繰り返しています。 結果は......全くできていませんねw #aimeetup

Nitor

@yuntian9999

这几天 X 上一直刷到 Jev。 最反常的一点是：它根本不写字。 TypeSafe AI 把 Jev 定义成第一款公开的 “System One Model”： 你给它一段状态 + 几个明确问题，它不生成长文，而是直接返回选择、评分、概率和置信度。 官方价格： $0.042 / 100万输入 tokens，输出免费。 TypeSafe 还声称，在它针对 System One 任务的工作流里，Jev 可以做到 20–200× 更快、40–400× 更便宜。 注意，这是厂商自己的特定任务测试，不是“全面吊打 GPT”。 所以我觉得它不是 GPT / Claude 的替代品。 更像是给 Agent 加一个“判断层”： GPT / Claude 写和规划 → Jev 判断 / 路由 → 代码执行 真正值得看的不是又一个新模型。 而是 AI 开始分工了。

Vincent.seo

@xunaoo

Jev 가 우리 로직을 대체할 수 있을까요? 아닙니다. 대체가 아니라 로직 안으로 들어앉습니다. 제 브라우저 에이전트에 붙여 봤습니다. 판단 한 단계가 2초에서 0.2초, 모델 호출이 열 번에서 두 번. 그런데 정확도는 6%p 낮습니다.

abdullah a | مبرمج

@xooox888

للمبرمجين 👨‍💻✨ Jev من TypeSafe AI يقدم فكرة مختلفة عن نماذج الذكاء الاصطناعي المعتادة. بدل التركيز على المحادثات وتوليد النصوص، تم تصميمه لمساعدة التطبيقات على اتخاذ قرارات واضحة وسريعة داخل النظام ⚙️🤖 مناسب للمهام المتكررة مثل التصنيف، التوجيه، والأتمتة.

WTR

@wtry1102

話題のJev。 いまいちよくわかってなかったので Astra君に解説ショート動画をつくってもらった なるほど・・・なるほど🙄

Frit🅾️ Pendej🅾️

@vwapster

I think Jev and Laya were the missing pieces to algorithmic trading. Sure you can hardcode gates and adjust variables, but being able to add a real time decision maker within your algorith is a game changer. I can smell and taste an early retirement.

VertrAI

@vertr_ai

The small-LLM fallback for typing is an important detail: Jev chooses actions, while text generation stays with an LLM. We featured your 1x demo, with credit, in our short explainer of that division of labor: x.com/vertr_ai/statu…

VertrAI

@vertr_ai

154ms vs 860ms is a useful routing result, scoped to these 200 synthetic classification cases—not a blanket ChatGPT comparison. We credited your latency and cost charts in our Jev video, alongside the game/browser demos: x.com/vertr_ai/statu…

Vadim Choi

@vadimchoi

$10k/month just to decide which agent should answer. not to answer. to decide. group chat + @mentions: fast, cheap, terrible UX. an LLM orchestrator: nice UX, 4-7s and $0.00046 a message. @typesafeai jev: 0.29s median, $0.00002. same decision, $20/month.

Ayush Jain

@urayushjain

Most AI models are built to generate. But Jev is trying something different: Don’t generate. Decide. Give it some context and a specific question, and it returns a structured decision with a probability. I found this interesting for things like model routing, agent workflows, tool safety and RAG. The idea is simple --Instead of calling a big LLM for every small decision, let your code handle the workflow and use a smaller model where human-like judgment is actually needed. Honestly, this feels like an interesting direction for AI systems.

つぼね👨‍💻

@tubone24

Jevに飽きちゃったのと、全然利用料が安いので、公開してみました。 雨の日はスピードで遊ぼう！！ jev-speed.tubone24.workers.dev

tshmieldev

@tshmieldev

Jev vs LLMS for clasifying tweets in your feed! Try it out: github.com/tshmieldev/sha… Make sure to drop a star!

Akshay 🚀

@akshay_pachaar

A faster, 100% open-source alternative to Jev. It’s called Laya. Both models solve the same problem. You give them unstructured state plus typed questions. Instead of generating a paragraph, they choose from answers you define and return probabilities your code can act on. But they make those decisions differently. Jev is a closed, hosted API. Its architecture and training are not public, but it works zero-shot and supports up to 255 options. Laya is Apache 2.0 licensed, runs locally, and uses open-weight encoder models. Here is how its pipeline works. → It routes by language first. Plain Python selects the English or multilingual checkpoint before inference. → It creates one sequence for each question. Every possible answer gets a `[MASK]` marker. The question and state provide the context needed to compare those options. → It scores instead of generating. A bidirectional encoder and small decision head assign one score to each option. Softmax turns those scores into a probability distribution. → It batches the questions. They run together in one model call, although every question still carries its own copy of the state. This removes network latency and token-by-token generation. In one independent test, Laya took about 35 milliseconds on a local GPU. Jev took roughly 380 milliseconds over the network. Laya also has no per-call API fee, keeps data on your infrastructure, and can run on older GPUs. But it is not a drop-in replacement for every workload. Jev is a hosted generalist that works well out of the box. Laya is a fast base you need to specialize. Laya makes the most sense when you have a stable answer set and representative training data. Fine-tune it for the task, calibrate its probabilities on held-out examples, and send uncertain decisions to a stronger model or human reviewer. Jev gives you stronger zero-shot decisions. Laya gives you speed, privacy, control, and no per-call fee after you teach it your task. The open-source repository is here: http://github.com/NandhaKishorM/laya If you want to self-host Laya, InstaCloud provides a really nice and easy way to do so. Check this out: https://www.instacloud.com/templates/laya If you're new to decision models, I wrote the full breakdown of how Jev. The article is quoted below.

Baris Terzioglu

@terzioglubrs

Agent memory is usually just an append-only Markdown file that grows forever, and most frameworks load the whole thing into context on every single run. That has been bothering me for a while, but never quite enough to fix it for our agents. Jev feels like it might be a low effort patch to this, by simply scoring each memory against the prompt first, then load only the relevant bits for that specific request.

Petru - Tech Driven

@techdrivenpetru

Yes but not with Jev. I used another classifier, locally, just as fast, just as good. Example: setup a "server" in python that loads the classifier model. Add a hook in Claude that fires on "pre-tool-use" and next time you ask Claude a random question like "how do I lint check a project?" and it tries to freelance and read your entire repo only to intoxicate itself and pollute its context, the classifier will slap its hand, say "no sir, you answer from knowledge" and deny the tool call. I tested this yesterday with success, but need to refine it as it misfires. Basically I was able to identify general queries, instances where I would ask something and Claude would rush ahead and run pip install without me asking or just write code instead of answering. A classifier is hypercheap compared to a regular LLM and would catch all of these. The model I used was DeBERT large. It's still stupid fast, I tested it on an Apple with M1 (regular) and you don't feel it running.

Prasenjit Sarkar

@stretchcloud

I keep seeing the same fix across agent stacks this week: teams are ripping the expensive model out of the middle of their decision loops. TypeSafe shipped Jev in early access on September 15. Not a chat model. They call it a System One model: hand it a state and typed options, it returns calibrated probabilities, no text generation at all. The mechanism is simple. Most agent loops burn a full LLM call on decisions that never needed generation: route to worker A or B, is this result relevant, approve or block this action. TypeSafe claims up to 200x faster inference and 400x lower cost on that class of decision. Ricker's own tests below land at 193x and 444x. What happened next is the real story. Within four days, Cognition's Jared Palmer shipped Kev, a LoRA adapter on Qwen2.5-0.5B trained in 1 hour 45 minutes on a MacBook Pro, with an API close enough to point TypeSafe's own SDK at it. Kev now scales up to a 9B version that trails Jev by about 4.5 points on held out evaluation. Laya-MLX arrived the same week targeting millisecond decisions on Apple Silicon. A community leaderboard, JevBench, already ranks a dozen of these models. The adoption signal convinces me this is not a toy. TanStack AI shipped a native decide() API for typed choices, scores and booleans. Beacon, an open source memory layer, uses Jev to score which coding sessions are worth turning into reusable lessons. Three teams, one primitive, inside a week. RouteLLM out of Berkeley showed back in 2024 that routing simple queries to a cheap model cuts cost over 85% while holding 95% of GPT4 quality, and production semantic routers report 40 to 90% savings today. What changed is that the router stopped being a side project and became a shipped, benchmarked model category with a name. The bottleneck this solves is real: every agent framework has a model sitting in a loop answering questions that never needed a sentence back. My read is that the decision layer becomes as standard a piece of the agent stack as the vector database became for retrieval, and whoever owns the default there owns a lot of the unit economics conversation for the next year of agent infrastructure. https://x.com/0xRicker/status/2101705843200721203

Stanislav Sorokin

@stas_sorokin_

1,000 AI papers sorted into 24 topics for $0.0585. Then Opus 5 graded the labels. @nutlope's Jev paper map went viral, but the pipeline never shipped and the eval was "still running". So I rebuilt both and opened them. The first judge run came back empty: Opus spent its whole budget thinking and answered nothing. Reasoning off, second run: it agreed with Jev on 85 of 100 papers, at 153x the cost and 1.9s against 57ms per paper. The 15 misses are not random. One number Jev already returns tells you which labels to recheck. Cheap models sort. Expensive models audit only what the cheap one flags. Repost if you classify anything at scale, because the eval rows are public and anyone can rerun them with their own judge in one command. Code in the reply.

内田勉 DirecTune.app β公開中

@sidodtv

なんか24時間後に Tiboリセットくるらしいので、今のうちに #directune の開発を進めるよ〜！ まずは、ゆっくり風動画解説で背景が自動作成されるようになったよ。試しに Jevの解説動画を作ったよ DrecTuneは、AIがアシストしてくれる動画生成・制作サービスだぞ‼️

さっと

@satto_sann

Jevでサクッと分類器は価値があるのかビルドトラップできそう

Andrew

@s4yonnara

Jev CEO, Diogo Almeida (ex-OpenAI): "It's not Claude Code because Claude Code is still part of the assistance era." He co-authored GPT-4, ChatGPT, InstructGPT and RLHF. Now he's building what comes next: 200x faster. 400x cheaper. A decision brain for AI agents. His argument is simple: frontier LLMs are insanely good at creating, but we're wasting them on thousands of tiny decisions: Which agent goes next? Is this relevant? Does this need approval? Those calls don't need generation. They need a model built to decide. In 18 minutes, Almeida explains why today's AI was built for assistance, not real automation, and why "tomorrow's AI will be for automation." This makes most $500 agent engineering courses look obsolete. Watch it today, then steal my 10-step guide to giving your agents a decision brain below ↓

s1rozha1

@s1rozha_

Jev hype is not about chat. it’s about decisions everyone is trying to compare it to ChatGPT / Claude / Cursor wrong frame Jev doesn’t need to write you a beautiful paragraph it needs to answer questions like: should this agent act now? which tool should run next? is this signal worth trading? is this market mispriced? should this wallet be ignored or watched? that is the interesting part prediction markets are basically endless small decisions under uncertainty not one giant “AI intelligence” moment thousands of tiny calls: price changed liquidity moved wallet entered news dropped odds lagged edge appeared edge disappeared you don’t want a slow model writing essays for that you want something cheap, fast, typed, and measurable that’s why Jev is interesting. not because it replaces frontier models because it makes the boring decision layer cheap enough to run everywhere

りょう｜AI駆動開発でのゼロイチマネタイズ

@ryo_hukugyo_pro

Jevをいかに使いこなすかが今後トークン節約や自動化をするAI社員を大量量産する上で、 めちゃくちゃ重要。

Richard Seroter

@rseroter

I'm reading up on Jev (typesafe.ai/blog/introduci…) and learning how it differs from "regular" classification ML models. I gave a bunch of articles to @googlecloud Gemini Enterprise and asked it for a breakdown. Helpful. Still digging in before going hands-on.