Text classification with JSON: design labels and validate outputs

Define labels around a decision

Write down the destination for each label before writing the question. “Billing” can mean a payment error, a subscription change, or any message containing a price. Those meanings lead to different routes. A useful taxonomy spells out both what belongs in a category and how to handle overlap.

These definitions are a proposed policy, not a universal taxonomy. A payment integration failure could be technical even though it mentions billing. Put that boundary in the option description if it matters to your team. If tickets regularly contain two independent requests, consider two questions or a multi-label workflow instead of forcing one category.

TypeSafe’s primitive guide recommends Choice for an unordered set of options and suggests an “other” or “none of the above” option when your list may not cover every input. The unclear option here gives the application a deliberate review path.

Keep evidence in context and policy in the question

Provide the facts needed to distinguish your labels. A message about a failed connection is easier to route when the context identifies the affected integration. Avoid adding unrelated customer history simply because it is available: every field should serve the judgment you are asking for.

{
  "message": "The payment integration fails to connect. Can you help?",
  "product_area": "integrations"
}

In the playground, add a Choice question with instructions such as “Which support queue should handle the problem in message?” Then define the four named options above. For a direct TypeSafe request, the corresponding question has this shape:

{
  "route": {
    "type": "choice",
    "instructions": "Which support queue should handle the problem in message?",
    "criteria": {
      "billing": "Charges, invoices, or subscription payment problems",
      "technical": "A feature or integration fails, including payment integrations",
      "sales": "Questions about a plan before purchase",
      "unclear": "Not enough detail to choose a support queue"
    }
  }
}

The key route identifies the answer for your code. TypeSafe documents that question IDs are not sent to the model, so the instructions must contain the complete question. The playground documentation explains the request and answer fields in more detail.

Validate the contract before taking action

There are two different checks: whether the response has the expected structure, and whether it made a good classification. Validate the first mechanically. Evaluate the second against reviewed examples. A syntactically valid JSON response can still assign the wrong ticket to a queue.

The following JSON Schema describes a small application-owned routing record. Your code constructs it after reading the model answer; it is not the full TypeSafe response schema.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "ticket_id": { "type": "string" },
    "category": { "enum": ["billing", "technical", "sales", "unclear"] },
    "needs_review": { "type": "boolean" }
  },
  "required": ["ticket_id", "category", "needs_review"],
  "additionalProperties": false
}

The JSON Schema project documents how enum limits accepted values and how required and additionalProperties control object fields. In this simple schema, missing fields, unknown categories, and unexpected extra fields fail validation.

Also verify that you received an answer for every requested question. Treat a timeout, an unreadable body, or a missing answer as a failed request. None of those outcomes should be silently converted into a legitimate business label.

Test boundaries and the review path

Create a small collection of tickets that people have labeled using the written policy. Include ordinary examples, overlapping requests, short messages, and cases with insufficient information. Keep expected labels separate from the context sent to the model.

• Check billing versus technical cases that mention the same payment terms.
• Check whether vague requests reach the review queue.
• Record disagreement by category instead of relying only on one overall accuracy number.
• Rerun the same evaluation after changing labels, descriptions, model versions, or routing policy.

A confidence value can help prioritize review, but its meaning depends on the model’s definition. TypeSafe’s confidence summarizes the distribution across options; it is distinct from an individual option’s probability. Use the confidence and calibration guide to design that part of the workflow.

Once the labels and error handling are explicit, try the context and Choice question in the playground. Read the actual result and adjust the policy where it is ambiguous. A single plausible answer is a starting point for evaluation, not evidence that the workflow is ready to automate.