Evaluating AI flows

A service that calls an LLM cannot be tested by comparing its output with an expected string - the model will phrase the same answer ten different ways. What it can be tested on is everything it promises regardless of phrasing: the structure of its output, the facts it must preserve, and the error rate it holds over real traffic. The tests below cover the services built in structured output and chat with memory, and the audit log measures what tests cannot see.

Prerequisites. Both example services deployed with their REST channels, and pip install pytest requests wherever the tests run.

Assert structure, not wording

The extraction service promises a contract - the three keys, their types, the facts from the input - and every part of that promise is assertable without ever comparing prose:

# -*- coding: utf-8 -*-

# stdlib
import requests

# Where the environment under test listens
base_url = 'http://localhost:17010'

def test_extract_order_structure():

    text = 'Hi, this is Jane Smith, I would like to order two ergonomic keyboards.'
    response = requests.post(base_url + '/example/extract-order', json={'text':text})

    assert response.status_code == 200
    data = response.json()

    # The contract - the keys and their types, regardless of the model's phrasing
    assert set(data) >= {'customer_name', 'product', 'quantity'}
    assert isinstance(data['quantity'], int)

    # The facts - stated in the input, so they must survive extraction verbatim
    assert data['customer_name'] == 'Jane Smith'
    assert data['quantity'] == 2

    # Note what is absent - no assertion on data['product'] equalling any exact
    # string, because "ergonomic keyboard" vs. "ergonomic keyboards" is the model's
    # phrasing, not the service's promise. Assert the substance instead:
    assert 'keyboard' in data['product'].lower()

def test_extract_order_missing_value():

    # A value the text does not state must come back as null - the skill says so,
    # and this test is what verifies the skill after every rewording.
    text = 'Please send ergonomic keyboards to John Doe.'
    response = requests.post(base_url + '/example/extract-order', json={'text':text})

    data = response.json()
    assert data['quantity'] is None

The second test is the important habit - every rule the skill states gets a test, so editing the skill's wording, or switching the model behind the connection, reruns the whole contract in seconds. The tests never name the model, so they are the evaluation harness for comparing models too - point the connection at a candidate, run the suite, read the verdict.

Replay a conversation

Memory is a behaviour across turns, so the fixture is a scripted conversation - each turn a request, with assertions on what the reply must still contain and on where memory must end:

def test_chat_remembers_within_one_user():

    url = base_url + '/example/support-chat'

    # A unique user id per test run keeps runs independent of each other -
    # a repeated id would bring history from the previous run into this one.
    import uuid
    user_id = 'test-' + uuid.uuid4().hex[:12]

    # Turn 1 states a fact, turn 2 asks for it back
    requests.post(url, json={'user_id':user_id, 'text':'My order 4711 has not arrived'})
    response = requests.post(url, json={'user_id':user_id, 'text':'What was my order number again?'})

    # The order number must be in the reply - the phrasing around it is free
    assert '4711' in response.json()['reply']

def test_chat_never_leaks_across_users():

    url = base_url + '/example/support-chat'

    import uuid
    alice = 'test-' + uuid.uuid4().hex[:12]
    eve = 'test-' + uuid.uuid4().hex[:12]

    requests.post(url, json={'user_id':alice, 'text':'My order 4711 has not arrived'})
    response = requests.post(url, json={'user_id':eve, 'text':'What was my order number again?'})

    # Eve's conversation has no turns - the number must not appear, regardless
    # of the rest of the reply
    assert '4711' not in response.json()['reply']

The isolation test is the one that matters most and the one no manual check keeps verifying - it pins the per-user memory promise from chat with memory permanently.

Measure what tests cannot see

Tests run a handful of curated inputs - production runs everything else. The audit log records every call either direction of the AI layer makes, so the evaluation of the deployed flow is a measurement, not a rerun:

• The retry loop in the extraction service makes malformed model replies visible as extra calls - an extraction averaging two attempts is a skill or model problem, quantified per day without any instrumentation.
• The built-in alert rules are the continuous part of the evaluation - error rates over 10% warn, over 25% the alert is diagnosed by an LLM, and average completion times have their own thresholds. A model swap that quietly degraded latency shows up there, not in the test suite.
• Per-call token usage, as token usage logs it, turns "the new prompt works better" into "the new prompt works better and costs 30% less" - both halves measured.

Failure behavior

These tests spend real tokens and inherit real flakiness, which changes how failures are read. A structural assertion failing once is the retry loop's job, and it failing repeatedly is a real regression in the skill, the model or the service - rerun before reverting. The conversation tests are immune to one source of flakiness by construction: fresh ids per run mean no state from a previous run can fail this one. And when the provider itself is down, every test fails at once with the connection's timeout - which is not a suite regression, it is a provider failure, and the approach under provider failures applies.

See also