X · by Nikhil Mudholkar
1,565 emails, six decision models
Perplexity v1.1 came out best overall, OpenAI was the fastest.
nikhil mudholkar
@nikhilmudholkar
I put OpenAI’s new Decisions API through the same 1,565-email benchmark I’ve been running on Jev, Perplexity and Clef. 6 models from 4 companies now. Perplexity v1.1 came out best overall 🚀 OpenAI was the fastest⏱️ But the confidence behavior is still the most interesting part to me. Jev was the only model that said “≥99% sure” hundreds of times without getting one of those answers wrong. Its the cocky guy in the class who is also right most of the time🥲 Clef has massive imposter syndrome! it's much better than its confidence suggests. And Perplexity changed a lot in just 4 days. v1 gave just 3 answers at ≥99% confidence. Four days later, v1.1 gave 323. It also got slightly more accurate and cut the price in half. kudos @AravSrinivas and @perplexity_ai for making such massive strides and shipping so quickly! There's no doubt now that @typesafeai has opened a pandora's box and as @CompleteSkeptic said, forked a new branch in the AI timeline with these decision models!
Oct 7, 2026 · 14 likesOpen on X
Nikhil Mudholkar put OpenAI's Decisions API through the 1,565-email benchmark he already ran on Jev, Perplexity and Clef. That makes six models from four companies. Perplexity v1.1 came out best overall and OpenAI was the fastest. He says the confidence behaviour is still the most interesting part.
Open the source
