# Evaluating AI flows

Test LLM services on structure and facts, and measure the deployed flow over the audit log.

A service that calls an LLM cannot be tested by comparing its output with an expected string - the model will phrase the same answer ten different ways. The tests check the properties that do not depend on phrasing: the structure of its output, the facts it must preserve, and the error rate it holds over real traffic. The tests below cover the services built in [structured output](https://zato.io/docs/ai/examples/structured-output.html) and [chat with memory](https://zato.io/docs/ai/examples/chat-with-memory.html), and the audit log measures what tests cannot see.

**Prerequisites.** Both example services deployed with their REST channels, and `pip install pytest requests` wherever the tests run.

## Assert structure, not wording {#assert-structure-not-wording}

The extraction service has a defined contract - the three keys, their types, the facts from the input - and every part of that contract can be asserted without ever comparing prose:

```python
# -*- coding: utf-8 -*-

# stdlib
import requests

# Where the environment under test listens
base_url = 'http://localhost:11223'

def test_extract_order_structure():

    text = 'Hi, this is Jane Smith, I would like to order two ergonomic keyboards.'
    response = requests.post(base_url + '/example/extract-order', json={'text':text})

    assert response.status_code == 200
    data = response.json()

    # The contract - the keys and their types, regardless of the model's phrasing
    assert set(data) >= {'customer_name', 'product', 'quantity'}
    assert isinstance(data['quantity'], int)

    # The facts - stated in the input, so they must survive extraction verbatim
    assert data['customer_name'] == 'Jane Smith'
    assert data['quantity'] == 2

    # Note what is absent - no assertion on data['product'] equalling any exact
    # string, because "ergonomic keyboard" vs. "ergonomic keyboards" depends on the
    # model's phrasing and is not part of the contract. Assert the substance instead:
    assert 'keyboard' in data['product'].lower()

def test_extract_order_missing_value():

    # A value the text does not state must come back as null - the skill says so,
    # and this test is what verifies the skill after every rewording.
    text = 'Please send ergonomic keyboards to John Doe.'
    response = requests.post(base_url + '/example/extract-order', json={'text':text})

    data = response.json()
    assert data['quantity'] is None
```

The second test is the important habit - every rule the [skill](https://zato.io/docs/ai/skills/index.html) states gets a test, so editing the skill's wording, or switching the [model behind the connection](https://zato.io/docs/ai/examples/self-hosted-models.html), reruns the whole contract in seconds. The tests never name the model, so they are the evaluation harness for comparing models too - point the connection at a candidate, run the suite, read the results.

## Replay a conversation {#replay-a-conversation}

Memory is a behavior across turns, so the fixture is a scripted conversation - each turn a request, with assertions on what the reply must still contain and on where memory must end:

```python
def test_chat_remembers_within_one_user():

    url = base_url + '/example/support-chat'

    # A unique user id per test run keeps runs independent of each other -
    # a repeated id would bring history from the previous run into this one.
    import uuid
    user_id = 'test-' + uuid.uuid4().hex[:12]

    # Turn 1 states a fact, turn 2 asks for it back
    requests.post(url, json={'user_id':user_id, 'text':'My order 4711 has not arrived'})
    response = requests.post(url, json={'user_id':user_id, 'text':'What was my order number again?'})

    # The order number must be in the reply - the phrasing around it is free
    assert '4711' in response.json()['reply']

def test_chat_never_leaks_across_users():

    url = base_url + '/example/support-chat'

    import uuid
    alice = 'test-' + uuid.uuid4().hex[:12]
    eve = 'test-' + uuid.uuid4().hex[:12]

    requests.post(url, json={'user_id':alice, 'text':'My order 4711 has not arrived'})
    response = requests.post(url, json={'user_id':eve, 'text':'What was my order number again?'})

    # Eve's conversation has no turns - the number must not appear, regardless
    # of the rest of the reply
    assert '4711' not in response.json()['reply']
```

The isolation test is the one that matters most and the one no manual check keeps verifying - it runs with the rest of the suite and confirms the per-user memory isolation described in [chat with memory](https://zato.io/docs/ai/examples/chat-with-memory.html).

## Measure what tests cannot see {#measure-what-tests-cannot-see}

Tests run a handful of curated inputs - production runs everything else. The [audit log](https://zato.io/docs/ai/llm/connections.html#the-audit-trail) records every call either direction of the AI layer makes, so the deployed flow is evaluated by measuring its production traffic:

- The retry loop in the extraction service makes malformed model replies visible as extra calls - an extraction averaging two attempts is a skill or model problem, quantified per day without any instrumentation.
- Per-call token usage, as [token usage](https://zato.io/docs/ai/examples/token-usage.html) logs it, turns "the new prompt works better" into "the new prompt works better and costs 30% less" - both halves measured.

## Failure behavior {#failure-behavior}

These tests spend real tokens and inherit real flakiness, which changes how failures are read. A structural assertion failing once is the retry loop's job, and it failing repeatedly is a real regression in the skill, the model or the service - rerun before reverting. The conversation tests are immune to one source of flakiness by construction: fresh ids per run mean no state from a previous run can fail this one. And when the provider itself is down, every test fails at once with the connection's timeout - which is not a suite regression, it is a provider failure, and the approach under [provider failures](https://zato.io/docs/ai/examples/provider-failures.html) applies.

## See also {#see-also}

- [Structured output](https://zato.io/docs/ai/examples/structured-output.html) - The extraction service these tests cover
- [Chat with memory](https://zato.io/docs/ai/examples/chat-with-memory.html) - The conversation service the replay tests cover