By JEV AI Model ·
An LLM evaluation rubric should describe observable differences between answers. Replace “rate the quality from one to five” with levels that explain what a reviewer would see at each point. Then test whether those levels produce useful, repeatable judgments on examples from your own task.
This guide uses support replies to show how to write a rubric, separate independent criteria, and review disagreements. The rubric is a worked design example. No evaluation scores or agreement rates are presented as measured model performance.
Choose one criterion per question
A reply can be friendly but incorrect, correct but incomplete, or complete but hard to follow. Combining those qualities into a single instruction makes a low score difficult to diagnose. Decide which qualities matter and ask about them independently.
For this example, use policy accuracy and resolution usefulness. Policy accuracy asks whether the reply is consistent with the supplied support policy. Resolution usefulness asks whether the customer gets an actionable next step. Tone can become a third criterion if it changes the product decision.
TypeSafe’s primitive documentation recommends focused judgments and composing independent factors in application code. A Jev Score question uses an ordered list of level descriptions. A Noul question suits a narrow yes/no condition, such as whether the reply asks for information the customer already supplied.
Write levels a reviewer can distinguish
Start with the lowest and highest acceptable descriptions, then add an intermediate level only when it serves a real distinction. Labels such as “poor,” “average,” and “excellent” leave the scoring policy implicit. The levels below define what is missing or present.
| Level | Policy accuracy | Resolution usefulness |
|---|---|---|
| 0 | Contradicts a relevant rule in the supplied policy | No usable next step, or a step unrelated to the request |
| 1 | No explicit contradiction, but a relevant condition is omitted | Relevant next step, with a necessary detail missing |
| 2 | Matches the relevant rule and its stated conditions | Clear next step with the details needed to carry it out |
These anchors still require enough evidence. If the state contains no support policy, the evaluator cannot reliably check policy accuracy. Treat missing evidence separately: add an evidence-availability check or route that case to review instead of interpreting the middle score as “unknown.”
Write one example at each level and one example near each boundary. Ask two reviewers to apply the descriptions independently where practical. Their disagreements can reveal an ambiguous policy before a model is involved. Refine the wording using those disagreements, then keep a separate evaluation set to check the revised rubric.
Turn the anchors into a Score question
Put the customer request, proposed reply, and relevant policy in the state. Use the question instructions to name the criterion and the fields to evaluate. In the playground, the levels are ordered from lowest to highest; their positions define the scale.
{
"policy_accuracy": {
"type": "score",
"instructions": "Evaluate reply against the relevant rules in policy for customer_request.",
"criteria": [
"Contradicts a relevant rule in the supplied policy",
"No explicit contradiction, but a relevant condition is omitted",
"Matches the relevant rule and its stated conditions"
]
}
}A returned score can fall between levels. TypeSafe’s answer reference describes the score as a position along the supplied levels, with a legend and a distribution across them. A value such as 1.4 should therefore be read against those anchors. It is not automatically a percentage correct or an assessment of every aspect of the reply.
Keep the individual criterion results when you combine them. A weighted total is a policy you choose; document its weights and any rule that overrides the total. For example, a policy contradiction may require review even when the reply is otherwise clear. Averaging ordered categories also introduces a measurement assumption, so use a total only when it helps the downstream decision.
Measure disagreements before automating
Run the rubric on examples that were not used to write its levels. Preserve the state, expected judgment, rubric version, returned model identifier, and result. Review disagreements by cause: missing context, unclear criterion, overlapping levels, or a model error despite a clear policy.
Research on LLM judges has identified position, verbosity, and self-enhancement biases. Zheng and colleagues’ MT-Bench and Chatbot Arena paper documents these limitations. Those findings do not establish how Jev performs on your rubric, but they give you useful failure cases to test rather than assume away.
- Compare a concise correct reply with a longer reply containing the same information.
- For pairwise comparisons, swap answer order and inspect changed judgments.
- Check whether irrelevant politeness masks an incorrect instruction.
- Review cases near a decision threshold, not only obvious successes and failures.
Choose acceptance criteria before inspecting all results. Depending on your workflow, you may care most about agreement within one level, missed policy contradictions, or the proportion sent to a reviewer. Report the sample size and the examples used so a good-looking result does not lose its context.
Version the scoring policy with the model
Changing a level description changes the evaluation task. Record the revision, explain why it changed, and rerun the same retained cases. Keep a fresh test set for decisions you did not anticipate while editing the rubric. If you change the model too, distinguish those changes when interpreting the result.
Use the existing examples to learn the UI, then build a Score question for one criterion from this guide. The confidence guide covers the separate decision of when a returned assessment should trigger human review.