Jev-like decisions from open LLMs

Anand Prasad

@theanandprasad

you can turn any low-cost open-source LLM into a Jev like fast decision model without training it. Here's how to do it: An LLM doesn't write text directly. At every step it scores every word in its vocabulary for how likely that word is to come next, and then it picks one. Those scores are called logits. Suppose you want to know whether a customer email is angry. The usual way is to ask the model, wait while it writes a paragraph, and then try to pull a yes or no out of the response. The trick is to never let it write anything. Give it the email content and the question, and end your prompt with "Answer:". Your full prompt will look something like this: [ Message: "Third time I've contacted you. Still broken." Is this customer angry? Answer yes or no only. Answer: ] The model reads the prompt once and scores every possible next word. You look at two of those scores. "yes" scores 8.1 "no" scores 5.9 Convert those two scores into probabilities and you get 90% yes and 10% no. (Use softmax function -- google it) Your code then uses that number directly. If it's above 0.8, escalate the ticket. Congrats, you have created your own Jev! Here’s why it works: 1. It's fast. Reading a prompt happens in one parallel pass. Writing is the slow part because the model writes one word at a time. Here the model never writes. 2. It's cheap. Output tokens are the expensive ones, and this produces none. 3. There's nothing to parse. You always get a fixed output, and never a paragraph. 4. You get a confidence level along with the answer. A 0.55 means the case should go to a human or a bigger model. A 0.98 means you can act on it. I tried running the code on llama 3.3 70b (via openrouter) and the accuracy was 21/22 (95%). Cost per query: $0.000013, latency: ~500ms