Multi-turn LLM conversations

The chat method is invoke with memory - the platform keeps each conversation's history under an id you choose, and every call sends the preceding turns along, so the model receives what was already said. Your service stays stateless, and the same conversation can continue from any server and across restarts.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class SupportChat(Service):

    name = 'demo.llm.support-chat'

    def handle(self):

        conn = self.llm['My Claude']

        # The chat id groups this user's turns into one conversation -
        # any string works, e.g. a user id or a session id.
        chat_id = self.request.payload['user_id']
        text = self.request.payload['text']

        response = conn.chat(text, chat_id=chat_id)

        self.response.payload = response['text']

The response is the same dictionary that invoke returns - text, usage and raw.

History storage

• Each chat's history is stored under its id, with the expiry the connection configures - one day by default, counted from the chat's last message. A chat that stays quiet longer than that starts afresh.
• History survives server restarts and is shared across all servers of an environment, so a conversation is never pinned to the server that started it.
• An unknown or expired chat id starts a new conversation - there is nothing to create first.

History trimming

Only the last max history turns of the conversation go out with each call - a turn is one user message plus the assistant's reply, and the limit is set on the connection, 20 turns by default. Older turns stay in the history until it expires but never leave the platform, which keeps long conversations from growing the token bill without a ceiling.

A named skill accompanies every call of the conversation:

response = conn.chat(text, chat_id=chat_id, skill='support-agent')

Concurrent calls

Two calls to the same chat at the same time would race over its history, so the platform serializes them - a per-chat lock makes the second call wait until the first one's reply is saved. Calls to different chats never wait for each other.

Without a chat id

A chat call without a chat_id loads nothing and saves nothing - it is exactly invoke. This lets one code path serve both cases, as in a service that receives an optional session id:

# An empty chat_id makes this a one-shot call
response = conn.chat(text, chat_id=chat_id_or_empty)

See also