EVALUATION

How to write an LLM evaluation rubric you can test

Build an LLM evaluation rubric with a copyable JSON template, scored support-reply examples, and boundary cases. Load the rubric in the free playground.

By JEV AI Model · · Updated

An LLM evaluation rubric should describe observable differences between answers. Replace “rate the quality from one to five” with levels that explain what a reviewer would see at each point. Then test whether those levels produce useful, repeatable judgments on examples from your own task.

This guide uses support replies to show how to write a rubric, separate independent criteria, and review disagreements. The rubric is a worked design example. No evaluation scores or agreement rates are presented as measured model performance.

Start with the complete rubric template, then adapt the criteria and test the boundary examples below.

Choose one criterion per question

A reply can be friendly but incorrect, correct but incomplete, or complete but hard to follow. Combining those qualities into a single instruction makes a low score difficult to diagnose. Decide which qualities matter and ask about them independently.

For this example, use policy accuracy and resolution usefulness. Policy accuracy asks whether the reply is consistent with the supplied support policy. Resolution usefulness asks whether the customer gets an actionable next step. Tone can become a third criterion if it changes the product decision.

TypeSafe’s primitive documentation recommends focused judgments and composing independent factors in application code. A Jev Score question uses an ordered list of level descriptions. A Noul question suits a narrow yes/no condition, such as whether the reply asks for information the customer already supplied.

Write levels a reviewer can distinguish

Start with the lowest and highest acceptable descriptions, then add an intermediate level only when it serves a real distinction. Labels such as “poor,” “average,” and “excellent” leave the scoring policy implicit. The levels below define what is missing or present.

LevelPolicy accuracyResolution usefulness
0Contradicts a relevant rule in the supplied policyNo usable next step, or a step unrelated to the request
1No explicit contradiction, but a relevant condition is omittedRelevant next step, with a necessary detail missing
2Matches the relevant rule and its stated conditionsClear next step with the details needed to carry it out

These anchors still require enough evidence. If the state contains no support policy, the evaluator cannot reliably check policy accuracy. Treat missing evidence separately: add an evidence-availability check or route that case to review instead of interpreting the middle score as “unknown.”

Write one example at each level and one example near each boundary. Ask two reviewers to apply the descriptions independently where practical. Their disagreements can reveal an ambiguous policy before a model is involved. Refine the wording using those disagreements, then keep a separate evaluation set to check the revised rubric.

For the fictional policy in the template below, these are illustrative reference judgments for policy accuracy, not measured model results:

LevelExample replyReason
0I refunded the extra charge without checking whether it was for the same order.Explicitly skips the verification required by the policy.
1I verified the duplicate charge and issued a refund.Omits the required 5–10 business day refund timeframe.
2I verified the duplicate charge for your order and issued a refund. It should reach your payment method in 5–10 business days.States verification and the required timeframe.

A useful boundary case is “I issued a refund. It should arrive in 5–10 business days.” The reply omits verification but does not explicitly say it was skipped. Under this rubric, that omission fits level 1 rather than level 0. These judgments assess what the reply says; they do not establish that a refund was actually processed.

Copy a complete rubric template

A complete support-reply evaluation template

Copy the context and both scoring questions, or load them in the playground. Review the inputs before running. This example uses a fictional refund policy.

{
  "state": {
    "customer_request": "I was charged twice for the same order. Can you refund the duplicate charge?",
    "support_policy": "Verify that both charges refer to the same order before refunding a duplicate. Tell the customer that the refund takes 5–10 business days.",
    "proposed_reply": "I verified the duplicate charge for your order and issued a refund. It should reach your payment method in 5–10 business days."
  },
  "questions": {
    "policy_accuracy": {
      "type": "score",
      "instructions": "How accurately does proposed_reply follow support_policy for customer_request?",
      "criteria": [
        "Contradicts a relevant rule in the supplied policy",
        "No explicit contradiction, but a relevant condition is omitted",
        "Matches the relevant rule and its stated conditions"
      ]
    },
    "resolution_usefulness": {
      "type": "score",
      "instructions": "How useful is the next step in proposed_reply for resolving customer_request?",
      "criteria": [
        "No usable next step, or a step unrelated to the request",
        "Relevant next step, with a necessary detail missing",
        "Clear next step with the details needed to carry it out"
      ]
    }
  }
}

Turn the anchors into a Score question

Put the customer request, proposed reply, and relevant policy in the state. Use the question instructions to name the criterion and the fields to evaluate. In the playground, the levels are ordered from lowest to highest; their positions define the scale.

A policy-accuracy Score question
{
  "policy_accuracy": {
    "type": "score",
    "instructions": "Evaluate reply against the relevant rules in policy for customer_request.",
    "criteria": [
      "Contradicts a relevant rule in the supplied policy",
      "No explicit contradiction, but a relevant condition is omitted",
      "Matches the relevant rule and its stated conditions"
    ]
  }
}

A returned score can fall between levels. TypeSafe’s answer reference describes the score as a position along the supplied levels, with a legend and a distribution across them. A value such as 1.4 should therefore be read against those anchors. It is not automatically a percentage correct or an assessment of every aspect of the reply.

Keep the individual criterion results when you combine them. A weighted total is a policy you choose; document its weights and any rule that overrides the total. For example, a policy contradiction may require review even when the reply is otherwise clear. Averaging ordered categories also introduces a measurement assumption, so use a total only when it helps the downstream decision.

Measure disagreements before automating

Run the rubric on examples that were not used to write its levels. Preserve the state, expected judgment, rubric version, returned model identifier, and result. Review disagreements by cause: missing context, unclear criterion, overlapping levels, or a model error despite a clear policy.

Research on LLM judges has identified position, verbosity, and self-enhancement biases. Zheng and colleagues’ MT-Bench and Chatbot Arena paper documents these limitations. Those findings do not establish how Jev performs on your rubric, but they give you useful failure cases to test rather than assume away.

  • Compare a concise correct reply with a longer reply containing the same information.
  • For pairwise comparisons, swap answer order and inspect changed judgments.
  • Check whether irrelevant politeness masks an incorrect instruction.
  • Review cases near a decision threshold, not only obvious successes and failures.

Choose acceptance criteria before inspecting all results. Depending on your workflow, you may care most about agreement within one level, missed policy contradictions, or the proportion sent to a reviewer. Report the sample size and the examples used so a good-looking result does not lose its context.

Version the scoring policy with the model

Changing a level description changes the evaluation task. Record the revision, explain why it changed, and rerun the same retained cases. Keep a fresh test set for decisions you did not anticipate while editing the rubric. If you change the model too, distinguish those changes when interpreting the result.

Use the existing examples to learn the UI, then load this rubric in the playground. The confidence guide covers the separate decision of when a returned assessment should trigger human review.