Jev confidence: when to act, ask or escalate

What the number is

confidence is a statistic computed from probabilities, which every Choice and Score answer also returns, so you can always check it or replace it. TypeSafe publishes the formulas.

Choice. With n options and p_max the probability of the chosen one, confidence = (p_max − 1/n) / (1 − 1/n). It measures how far the top option sits above an even split. Only the top probability counts, so (0.6, 0.3, 0.1) and (0.6, 0.2, 0.2) both give 0.40. In TypeSafe's ticket example, returns gets 0.61 of three departments and billing 0.35, and TypeSafe prints a confidence of 0.42 (these rounded probabilities give 0.415).

Score. Levels are ordered, so probability on a neighbouring level costs less than probability on a distant one. With three levels, (0, 0.5, 0.5) gives 0.25, a model torn between two adjacent levels, while (0.5, 0, 0.5) gives 0.00, torn between opposite ends. TypeSafe's bug report scored (0, 0.57, 0.43), which gives 0.35.

TypeSafe calls this "one reasonable way to summarize a distribution, not the only one". Two alternatives it suggests trying: the top probability on its own, which reads as "how likely is this option" but means different things with 2 options and with 10, so set it per question; and the ratio of the top probability to the second, for decisions that come down to two candidates.

// Choice: how far the top option sits above an even split.
function choiceConfidence(p: number[]): number {
  const n = p.length;
  return (Math.max(...p) - 1 / n) / (1 - 1 / n);
}

// Score: spread around the most likely level, against an even spread.
function scoreConfidence(p: number[]): number {
  const n = p.length;
  const m = p.indexOf(Math.max(...p));
  const spread = p.reduce((s, pi, i) => s + pi * Math.abs(i - m), 0);
  const even = p.reduce((s, _, i) => s + Math.abs(i - (n - 1) / 2), 0) / n;
  return Math.max(0, 1 - spread / even);
}

// Noul: no field is returned; this puts it on the same scale.
const noulConfidence = (yes: number) => Math.abs(2 * yes - 1);

Sources: How confidence is calculated, Choice, Score.

Why a Noul has no confidence

A Noul answer is one probability over two outcomes, so the number is the answer and the certainty at once. If you want Nouls and Choices on the same scale, use the distance from 0.5: |2p − 1|. firehose-judge does exactly this so that one review rule covers all three types.

If you threshold a Noul directly, TypeSafe's advice is: use 0.5 when yes and no are equally easy to act on. Raise the line when acting on a false yes is expensive, such as paging someone or issuing a refund. Lower it when missing a true yes is expensive, such as a safety issue. Send the middle to a person. Its support-routing example uses YES = 0.8 and NO = 0.2, with everything between going to review. If reviewers see too many cases, narrow the gap. If too many wrong routes get through, widen it.

Sources: Reading a Noul, firehose-judge src/jev.ts.

Calibrated across many answers, not for one

TypeSafe trains Jev with what it calls reinforcement learning for calibrated decisions. The target: across many answers, outcomes given 0.2 happen about 20% of the time and outcomes given 0.8 about 80%. That is a statement about groups. A confidence of 1.0 on a Score "describes the model's answer, not a guarantee that the answer is correct", and a reworded description that raises confidence hasn't been shown to be better until you check it against labels.

Calibration is also measured on someone's data, not yours. Bouncer publishes a per-question table from its own fixtures. Its destructive question was right 92% of the time over 26 answers, but its 0.8–0.9 bin held 2 answers and got 1 right. Small bins swing. Bouncer's bar for turning a question on is 85% accuracy among answers at 0.8 confidence or higher, compared on the exact ratio, because 11 of 13 is 84.6% and would round to a passing-looking 85%.

Sources: AI primer, System One, Reading a Score, Bouncer README.

Bouncer's figures are the authors' run, as of Oct 2026; the README table updates with each calibration run.

What a 0.98 means for a judge is on Jev as a judge.

Act, confirm or route

TypeSafe's starting pattern splits confidence into three ranges. High: act automatically. Medium: ask the user to confirm, flag it for review, or gather more information first. Low: don't act; route it to a person, ask for clarification, or fall back to another system. A single system usually needs more than one line, because a wrong read costs more for some actions than for others.

const action = response.answers.action;

if (action.confidence < 0.5) {
  // Genuinely unsure: don't guess.
  return routeToHuman(message);
}
if (action.choice === "check_balance") {
  // Low stakes: the wrong screen is recoverable.
  return showBalance(accountId);
}
if (action.choice === "approve_transfer") {
  // High stakes: even a confident read is confirmed.
  return action.confidence > 0.9
    ? confirmThenExecute(accountId)
    : askUserToConfirm(accountId);
}

In TypeSafe's version, the transfer is confirmed even above 0.9; the threshold only decides how it is confirmed. Its confidence-routing pattern uses the same idea with a 0.6 floor and 0.85 for a transfer. And not every answer needs a gate. If all you need is the best option, take the top one; TypeSafe's agent-skill page warns against "using confidence thresholds everywhere".

Sources: Thresholds scale with risk, Confidence-gated routing, agent skill: common issues.

The lines TypeSafe's cookbooks used

Each of these numbers was chosen for one dataset and one model version. They are examples, not defaults.

The lines builders drew

These are each author's own settings, from their repositories or posts. We haven't re-run them.

Set the line from the cost of a mistake

A threshold is a trade between two costs: acting on a wrong answer, and sending a case to someone who didn't need to see it. For a Noul you act on as a yes, the average cost of acting is (1 − p) × the cost of a wrong yes; the cost of review is whatever review costs. Acting is cheaper when p > 1 − review cost ÷ cost of a wrong yes. The same reasoning on the other side gives the line below which a "no" is safe: p < review cost ÷ cost of a missed yes. Between the two lines, a person decides.

With hypothetical costs, chosen only to show the arithmetic: if a wrong automatic refund costs as much as 20 reviews and a missed refund request as much as 4, act on yes above 0.95, treat it as no below 0.25, and review in between.

Our arithmetic; the costs are hypothetical.

This only works if p means what it says on your data, so check it against labels first. For a Choice or Score, confidence is not the probability of being right. Replace (1 − p) with the error rate you measured at that confidence in your labelled sample. jeval automates this: it reads your logged answers and human corrections, plots the expected cost for every candidate line, and writes out the one to deploy. Its example report recommends moving a line from 0.60 to 0.75, but that report is built from a synthetic demo log, not a real deployment.

Sources: Noul thresholds, jeval README.

Tune it in shadow

1. Log, don't act. Run Jev beside the current path and record every answer, with nothing changed. winnow has WINNOW_MODE=shadow ("run it for a week without trusting it"), Bouncer ships watching and logging, and 0xCodila ran a Grok Bot router in shadow and read the logs before switching it on.
2. Label a sample spread across the range. winnow's replay tool draws 100 blocks stratified by the judge's probability, so the middle isn't under-sampled.
3. Plot confidence against accuracy, per question. That is TypeSafe's own instruction for testing thresholds. firehose-judge serves the review share and human agreement at 21 thresholds, so you can see what a stricter bar costs in human hours.
4. Set one line per action, from the cost of a mistake (above).
5. Pin the model version once the lines are set. The response's model field names the version that answered (as of Oct 2026, jev-1.13.0).
6. Keep questions and thresholds in one file, where a reviewer can find them.
7. Re-tune when something changes: the model, a question's wording, or the classifier behind the same API. jev-antispam-bot's README says to retune SPAM_THRESHOLD after switching.
8. Watch both directions. TypeSafe's checklist: a line set too high causes false negatives, one set too low causes false positives, and sometimes the question needs to be more specific instead.

Sources: How to build: route on uncertainty, agent skill: common issues, winnow, firehose-judge, 0xCodila, models.

A lane for the unsure ones

Each of these acts on the confident answers and hands the rest to a person, a larger model or a do-nothing default. Hassan sent the emails under 95% confidence to Kimi K3, and firehose-judge puts posts Jev won't commit on in a lane marked "needs a human".

GitHub·Security and abuse

firehose-judge

The live Bluesky firehose, judged post by post, with a lane for humans.

LMLeo Mata

0~$0.00003

GitHub·Social feeds

LinkedIn NoSlop

A Chrome extension that blurs low-value LinkedIn posts.

Ssushrutb17

0

GitHub·Security and abuse

Jev Anti-Spam Bot

A Telegram bot that deletes only the spam Jev is sure about.

NKNikita Kolmogorov

16

Hassan

@nutlope

Jev + Kimi K3 for fraud detection! TLDR: Jev classified 100 emails in 1.42 seconds, then I routed the uncertain cases to Kimi K3. The full pipeline got 96/100 correct for only ~$0.07. Video is not sped up, check out the live run! Here was my process: I gave Jev 100 emails to classify (a mix of 50 legit & 50 fraudelent emails). It classified all of them in 1.42 seconds. An underrated feature about Jev is it will give you the confidence score for a classification, so I routed any prediction under 95% confidence to Kimi K3 to be fully sure. 31 emails fell below that threshold. After routing those to Kimi K3, the combined pipeline reached 96% accuracy. The full run took 16 seconds & ~$0.07 in inference costs: - $0.068 from Kimi K3 on @togethercompute - $0.003 (1/3 of a cent) from Jev on @typesafeai. I think this is a really interesting pattern: use a fast specialized model like Jev for the narrow task, then route the uncertain cases to a larger LLM. I feel like this kind of approach could be a game changer for use cases like fraud or anything realtime. You can use the speed & low cost of Jev while having a larger LLM as a fallback to ensure high accuracy.

X·Security and abuse

Pick

Fraud detection with Jev and Kimi K3

Emails

100 in 1.42 s

Correct

96/100

Cost

~$0.07 pipeline ($0.003 Jev, $0.068 Kimi K3)

GitHub·Search

SemDecide

grep for meaning and jq for judgment, with a Unix exit code.

SSSharvil Saxena

76

GitHub·Security and abuse

jevmod

Moderation with a probability per category and thresholds you set.

OHOmar Hernandez

1

Akshay 🚀

@akshay_pachaar

GitHub·Benchmarks and evals

jev-as-judge

Traces

10 frozen

Questions

4 in one request

NO1ennn

@N01ennn

this is pure f*cking treasure A Stanford AI research group finally drew the perfect RAG system: retrieval, Jev and agents in one loop, and it fixes the 3 things that break every RAG app: > the LLM reads 20 passages when only 3 matter > it answers questions your docs can't answer > it trusts whatever text it retrieves here's how it runs: > a lead agent sends the query > hybrid search pulls the top 20 candidates, dense + keyword > ONE Jev request scores all 20 + 2 gates: answerable? injection? > only passages above 0.6 reach the writer agent > a second Jev call checks every claim against its source > grounded answer, with citations and when the docs don't have it: > answerable fails, the writer never runs > a researcher agent rewrites the query and retries once > still nothing? "not in the docs". zero tokens spent on a guess retrieval casts the net. Jev decides what's real. agents do the work save this before you build your next RAG

X·Search

A RAG loop where Jev decides what is real

Candidates scored per request

20

Passage threshold

0.6

Run it in shadow first

Judge everything, log it, change nothing, then read the logs. winnow's README calls its shadow mode a week of running it without trusting it, and Bouncer ships watching and logging rather than blocking.

GitHub·Context and memory

winnow

A context sieve for Claude Code: Jev judges each tool result before it lands.

GDGhaleb Dweikat

103

GitHub·Security and abuse

Bouncer

A judgment layer that checks every Claude Code tool call against your policy.

CClownware

1~$0.04/day

codila

@0xCodila

Jev + GrokBot is the best AI agent system I’ve built in my life It just made my setup CHEAPER and FASTER than what 95% of people are running... setup takes literally 7 minutes: prompt → GrokBot → Jev decision → GrokBot execution → result step 1 → open @typesafeai , create API key (keep it off chat paste) step 2 → tell Grok Bot: store TYPESAFE_API_KEY in the secure field step 3 → prompt Grok Bot: install typesafe-sdk on Agent Computer + smoke system_one (Choice) step 4 → tell Grok Bot: build the usage lab (router, dry-run, config, logs) - or clone Github below step 5 → add skill jev-usage-router: before browser / research / retry / extra bot → call the router, honor action step 6 → stay shadow first, read logs, then active when you trust it - kill switch: bypass jev or enabled: false step 7 → flip active: GrokBot obeys route - Jev decides - GrokBot executes - humans control irreversible actions the result: Jev + GrokBot the best and fastest agent running directly on your computer rn, I’ve already tested it on routine tasks - and the results are genuinely incredible You can come up with endless ways to use Jev + GrokBot - but the most important thing is to install it as soon as possible Copy this 2028 setup, explore my repo below - then read the full Jev deep dive ↓

X·Routing and model choice

A usage router for Grok Bot

Tools that find the line

Calibration checks and threshold pickers that work on your labelled answers. jeval turns what a mistake costs into the point where a person should take over.

GitHub·Benchmarks and evals

jeval

Works out what a Jev confidence score is actually worth.

Hhope

22

Skill·Benchmarks and evals

tenbin

Split a judgment into Choice, Score and Noul, lint it, then measure it.

SIShingo Imota

4

GitHub·Benchmarks and evals

jev-calibrate

Tune your Jev questions against your own labels, then confirm on held-out data.

Ssmkrv

32

GitHub·Benchmarks and evals

jev-align

Turns human labels into calibrated Jev classifiers with GEPA.

SSutro

305

GitHub·Benchmarks and evals

jev-evaluation

Nine experiments and 28 predictions, all fixed before any data.

WKWill Kelly

0123,805

GitHub·Benchmarks and evals

daf-jev

A Python toolkit: question builders, confidence gates and calibration.

DADaniel Ari Friedman

6

Where to go next

Confidence is only as good as the question behind it. If answers come back unsure across the board, check the wording first: Jev question types. The same gates, applied to labels, are on Jev for classification. Routing each turn to a cheaper or larger model on Jev's read is on the Jev router page. Your first call, and where the confidence appears in it, are on how to use Jev.