Chat with memory

A support chat where each user has their own conversation - the model receives this user's earlier turns and never another user's, history survives restarts, and a skill supplies the assistant's standing instructions without entering any conversation's history.

Prerequisites. An LLM connection named Support LLM and a skill named support-agent - both are defined under the configuration and the skill.

The service

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class SupportChat(Service):
    """ A per-user support chat - each user_id is its own conversation.
    """
    name = 'example.ai.support-chat'

    input = 'user_id', 'text'
    output = 'reply'

    def handle(self):

        conn = self.llm['Support LLM']

        # The user's own id is the chat id - this is the whole isolation mechanism.
        # Two users can never share a conversation because they never share an id.
        chat_id = 'support.' + self.request.input.user_id

        response = conn.chat(
            self.request.input.text,
            chat_id=chat_id,
            skill='support-agent',
        )

        self.response.payload.reply = response['text']

Run the service

Call the service twice for the same user:

curl localhost:17010/example/support-chat -d '{"user_id":"alice","text":"My order 4711 has not arrived"}'
curl localhost:17010/example/support-chat -d '{"user_id":"alice","text":"What was my order number again?"}'

The second reply names order 4711 - the model received the first turn along with the second question. Now call it as another user:

curl localhost:17010/example/support-chat -d '{"user_id":"bob","text":"What was my order number again?"}'

The model has nothing to answer from - Bob's chat_id is support.bob, a conversation with no turns in it yet. That is the expected output: memory per user, not per service.

History trimming

The connection below sets max_history_turns to 5, so each call sends at most the last 5 turns - a turn is one user message plus the assistant's reply. Ask ten questions numbered one to ten, then ask:

curl localhost:17010/example/support-chat -d '{"user_id":"alice","text":"What was my first question?"}'

The model cannot answer - questions one through five have been trimmed from what goes to the provider. They are still stored, and raising max_history_turns on the connection makes the next call send more of them - the semantics are under history trimming.

Expiry

With chat_expiry of 3600 below, a conversation quiet for an hour starts afresh - the expiry semantics are under history storage.

The skill

The skill accompanies every call, as use a skill describes - config/repo/skills/support-agent/SKILL.md:

---
name: support-agent
description: How to answer customer support questions
---

You are a support agent for an electronics retailer.

* Answer in the customer's language.
* Keep replies under three sentences.
* When you do not know, say so and point to support@example.com.

The configuration

The enmasse YAML - the connection, with its key in an environment variable, and the REST channel the curl calls above go through:

llm:
  - name: Support LLM
    model: gpt-4o-mini
    address: https://api.openai.com/v1
    secret: Zato_Enmasse_Env.Support_LLM_Key
    max_history_turns: 5
    chat_expiry: 3600

channel_rest:
  - name: example.ai.support-chat
    service: example.ai.support-chat
    url_path: /example/support-chat

Failure behavior

A provider outage mid-conversation raises an exception from chat and the user's message is not written into the history - the conversation is never left with a question the model never saw. The failed call is in the audit log and retrying the same request continues the conversation exactly where it was. Two simultaneous messages from the same user never race either - calls to one chat_id are serialized, as described under concurrent calls.

See also