X · by Stanislav Sorokin
Jev against Claude Haiku 5.5 on 262 real decisions
Jev got 204 right and Haiku 5.5 got 177.
Stanislav Sorokin
@stas_sorokin_
Jev raised $870M at $7.5B yesterday. On 262 of our real decisions it beat Claude Haiku 5.5, 204 to 177. TypeSafe's decision model got that valuation weeks after launch, in the same week Anthropic shipped Haiku 5.5 at $0.10 per million input tokens. So we checked whether a cheap general model can do the deciding. Haiku 5.5 got the labeled holdout cases from 17 decision systems we run every day, with the same instructions, word for word. → Right picks: Jev 204/262, Haiku 5.5 177/262 → By system: Haiku won 1, tied 5, lost 11 → Cost of the 262 calls: $0.058 on Haiku, $0.022 on Jev (2.7x) → Median answer: 1.16 s vs 375 ms → Biggest gap: "which builder gets this task" 6/12 vs 10/12 The twist: the one system Haiku won is the one that asks for judgment. "Did the agent really finish the task": Haiku 11/11, Jev 9/11. One caveat: those instructions were tuned on Jev for weeks. Haiku read them cold. Would you put a general model or a decision model behind your agent's routing? Repost this now, because "Haiku is cheap enough for everything" is turning into a default nobody measured. Follow @stas_sorokin_ for numbers from AI systems running in real production.
Oct 10, 2026 · 1 likesOpen on X
Stanislav Sorokin gave Claude Haiku 5.5 the labeled holdout cases from 17 decision systems his team runs every day, with the same instructions as Jev. Jev got 204 of 262 right and Haiku 5.5 got 177. The 262 calls cost $0.022 on Jev and $0.058 on Haiku, and the median answer took 375 ms against 1.16 s. Haiku won the one system that asks whether the agent really finished the task, 11 of 11 against 9 of 11. He notes that the instructions were tuned on Jev for weeks.
Open the source
