GitHub · by Akshay Pachaar

jev-as-judge

A refund agent graded by Jev, recorded as an Opik experiment.

Akshay 🚀

@akshay_pachaar

Sep 21, 2026 · 393 likesOpen on X

Akshay Pachaar’s worked example of Jev as the judge in an evaluation loop. Ten frozen support traces — a refund request, the policy, the tool calls and the agent’s final answer — are replayed, checked by plain code first, then sent to Jev as four questions in one request. The answers become separate Opik metrics, so the judgments sit in an experiment you can compare runs against.

The rubric is the interesting part. Three of the four questions are Noul: are all factual claims supported by the evidence, does the reply address the request, and are completed-action claims backed by a successful tool result. That third one is the whole reason the example exists: an agent that looks up an order and then says “Done, I have issued your refund” passes a deterministic check on the lookup and fails on the claim. The fourth is an ordered Score for helpfulness, normalised from its 0–2 rubric to 0–1.

The repository is careful about what the numbers mean, which is rarer than it should be. A grounded value of 0.98 is the probability of a yes to the whole proposition, not the share of the answer that is grounded. Score confidence summarises the spread across the rubric levels; it is not the probability that the judge is right. The 0.2 and 0.8 verdict boundaries and the 0.6 confidence threshold are called illustrative and uncalibrated, and the included expected verdicts are marked as tutorial annotations rather than collected labels.

Failures never become passes. An authentication error is not retried, an invalid response is marked judge_error, routed to review and exits non-zero, and the demo mode’s probabilities are labelled DEMO-hand-authored-not-Jev so nobody mistakes the offline run for evidence about the model. One of the ten traces is an attempt to talk the evaluator round, and the rubric answers it by telling Jev to treat the trace as data — which the author says reduces ambiguity but is not a proven defence against injection.

It runs offline with no dependencies and no network, and only reaches for Opik once you install the extra and configure a workspace. Written up as an article on X; the repository carries the draft, the figures and the validation notes alongside the code.