By JEV AI Model ·
A classifier confidence score is useful only when you know what it measures. A class probability, a certainty statistic, and an observed accuracy rate answer different questions. Before turning any of them into an automatic action, define the number, check it against reviewed examples, and decide what a wrong action costs.
This guide separates those concepts, explains calibration with a small numerical example, and shows how to compare review thresholds. Every numerical example below is synthetic and illustrates arithmetic, not measured Jev performance.
Read the model’s definition first
In Jev, a Noul answer provides the probability of “yes” as a number from zero to one. Choice and Score answers provide a distribution across the supplied options or levels, plus a separate confidence value. TypeSafe defines that confidence as a summary of the distribution’s shape: concentrated distributions indicate more certainty, while flatter distributions indicate less.
The official confidence documentation explicitly distinguishes these fields. Do not assume that a Choice confidence of 0.8 means the selected option has probability 0.8. Inspect probabilities when you need the probability assigned to a particular option.
| Value | Question it addresses | What it does not establish |
|---|---|---|
| Noul probability | How likely is the proposition to be true? | The cost of acting on it |
| Choice option probability | How much probability is assigned to this option? | Observed accuracy on your dataset |
| TypeSafe confidence | How concentrated is the answer distribution? | An interchangeable value for one option’s probability |
| Measured accuracy | How often did predictions agree with reference labels? | How well every confidence range is calibrated |
A Noul value near 0.5 means similar probability for yes and no. It does not mean a medium quantity or skill level. Use an ordered Score question when the task is to place something on a scale.
Calibration compares groups of predictions with outcomes
The scikit-learn calibration guide describes a well-calibrated binary classifier this way: among cases assigned positive-class probabilities near 0.8, roughly 80% should actually belong to the positive class. This is a statement about a collection of comparable cases, not a guarantee for one item.
Imagine 20 reviewed messages assigned urgency probabilities near 0.8. If 16 are urgent under your labeling policy, the observed positive fraction is 16 / 20 = 80%. If only 10 are urgent, the same group has an observed fraction of 50%, suggesting overconfidence for that group. Twenty cases are a small teaching example, so neither result is a precise estimate of future performance.
For a real check, retain predicted probabilities and independent reference labels. Group predictions into probability ranges, compare each range’s average prediction with its observed outcome frequency, and report how many examples each range contains. Inspect important subgroups, such as language or ticket category, when your data supports doing so.
Do not label the TypeSafe confidence statistic as a calibrated correctness probability merely because it lies between zero and one. You can still measure error rates at different confidence ranges, but the mapping is something to evaluate for your task.
Choose thresholds around an action
Predicting a category and deciding whether to act are separate steps. The scikit-learn threshold guide makes this distinction explicit and recommends selecting thresholds for the application’s objective. A threshold suitable for a reversible queue assignment may be unsuitable for an action that changes customer data.
Define the review policy before optimizing a number. For a support workflow, you might send low-certainty or unclear tickets to a reviewer and automatically route only the remaining cases. There is no universal confidence threshold that makes this safe or accurate across datasets.
Compare the error rate of automatic decisions with coverage: the fraction of all cases you handle automatically. Suppose a synthetic evaluation set contains 50 tickets. A candidate threshold routes 20 automatically, and 17 of those routes match the reference labels. Coverage is 20 / 50 = 40%; accuracy among automatic routes is 17 / 20 = 85%. Both numbers matter. A higher threshold might improve that subset’s accuracy while sending more work to people.
function routeTicket(answer, reviewedPolicy) {
if (answer.choice === "unclear") return "human_review";
if (answer.confidence < reviewedPolicy.minConfidence) {
return "human_review";
}
return answer.choice;
}The policy’s minConfidence must come from your evaluation and the cost of mistakes. Keep timeout handling and malformed responses outside this logic; a missing prediction should not fall through into an automatic route.
Retest when inputs, labels, or models change
Use a validation set to choose a threshold and a separate test set to estimate the behavior of the chosen policy. Reusing the same examples for training, repeated rubric edits, threshold selection, and final reporting makes it harder to tell whether the policy generalizes.
After a model or policy change, rerun retained cases and sample new traffic for review. Changes in the number of options, level definitions, or input mix can change how a confidence range relates to errors. Preserve the returned model identifier and question version with evaluation results.
For an end-to-end design, combine this process with the JSON classification contract and the rubric guide. Then inspect a real Jev result in the playground, keeping the model’s numbers separate from the application policy you build around them.