Structured output

Get JSON out of a model that downstream code can rely on - parsed, validated and retried.

A model asked for JSON returns JSON most of the time - and prose around JSON, markdown fences or a missing field often enough to break anything that assumes otherwise. The service below extracts order data from free-form emails and returns a dictionary that downstream code can rely on, because everything the model returns is parsed, validated and retried before it is returned to any caller.

Prerequisites. An LLM connection named Extractor LLM - any backend from self-hosted models works - and the skill below.

The skill

The skill states the format contract. Keeping the format rules in a skill means tightening them later is a file edit, not a redeployment - config/repo/skills/order-extractor/SKILL.md:

---
name: order-extractor
description: Extracts order data from free-form text as strict JSON
---

You extract order data from customer emails.

Respond with a single JSON object and absolutely nothing else - no explanations,
no markdown fences, no text before or after it.

The object has exactly these keys:

* "customer_name" - a string, the customer's name
* "product" - a string, what they are ordering
* "quantity" - an integer, how many they want

When a value is not stated in the text, use null for it.

The service

# -*- coding: utf-8 -*-

# stdlib
import json

# Zato
from zato.server.service import Service

# The keys the model's reply must contain - the same contract the skill states
_required_keys = ('customer_name', 'product', 'quantity')

# How many times a malformed reply is retried before the service gives up
_max_attempts = 3

class ExtractOrder(Service):
    """ Extracts structured order data from free-form text.
    """
    name = 'example.ai.extract-order'

    input = 'text'
    output = 'customer_name', 'product', 'quantity'

    def _parse_reply(self, reply):
        """ Returns the reply as a dict, or None when it is not the JSON we asked for.
        """

        # Models sometimes wrap JSON in markdown fences or prose - taking the slice
        # between the first { and the last } recovers the object from both cases.
        start = reply.find('{')
        end = reply.rfind('}')

        if start == -1 or end == -1:
            return None

        try:
            data = json.loads(reply[start:end + 1])
        except ValueError:
            return None

        # A reply that parses but misses a key breaks callers too
        for key in _required_keys:
            if key not in data:
                return None

        return data

    def handle(self):

        conn = self.llm['Extractor LLM']
        text = self.request.input.text

        for attempt in range(_max_attempts):

            response = conn.invoke(text, skill='order-extractor')
            data = self._parse_reply(response['text'])

            if data is not None:
                self.response.payload = data
                return

            self.logger.info('Malformed extractor reply, attempt %d: %s', attempt + 1, response['text'])

        raise Exception(f'No valid JSON after {_max_attempts} attempts')

Run the service

curl localhost:11223/example/extract-order -d '{"text":"Hi, this is Jane Smith, I would like to order two ergonomic keyboards."}'

Expected output:

{"customer_name": "Jane Smith", "product": "ergonomic keyboard", "quantity": 2}

The REST channel:

channel_rest:
  - name: example.ai.extract-order
    service: example.ai.extract-order
    url_path: /example/extract-order

Failure behavior

The retry loop is the failure path - a reply wrapped in prose or missing a key is logged with its attempt number and asked again, and the caller receives either a valid object or the final exception. Each attempt is a separate call in the audit log, so an extraction that routinely needs two attempts is visible there as double traffic - a measured signal that the skill's wording needs tightening or the model needs changing. Each retry's cost is in the response's usage dictionary - see token usage.

See also

FeatureWhat it does
SkillsThe instruction files that spell out the format contract
Token usageWhat each attempt of the retry loop costs
Evaluating AI flowsStructural assertions over responses like this one