Calling LLMs from clinical services

This page builds a service that turns an incoming admission event into a plain-language ward notification.

The service calls the model through a named LLM connection. The connection stores the provider protocol, model, address, credentials and runtime limits outside the service code, and it determines where the prompt is sent.

Before you begin

You need:

• A running Zato environment. The Docker installation guide sets one up.
• A REST channel that invokes your service.
• An LLM connection named Clinical LLM, created under Create the connection below.

Call a model from a service

The REST channel invokes the service with the admission event. The service sends the selected name, patient class, location and event time to the configured model:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

# #####################################################################
# #####################################################################

class WardNotification(Service):
    """ Turns an admission into a ward notification.
    """
    name = 'clinical.ward-notification'

    input = 'family_name', 'patient_class', 'location', 'event_time'

    def handle(self) -> 'None':

        # Read the admission event received by the REST channel ..
        data = self.request.input

        # .. select the fields needed by the notification ..
        family_name   = data.family_name
        patient_class = data.patient_class
        location      = data.location
        event_time    = data.event_time

        # .. build the prompt sent to the model ..
        prompt = (
            'Write a one-sentence ward notification for this admission: '
            f'patient {family_name}, class {patient_class}, '
            f'location {location}, at {event_time}.'
        )

        # .. invoke the named connection ..
        connection = self.llm['Clinical LLM']
        response = connection.invoke(prompt)

        # .. and log the returned notification.
        notification = response['text']
        self.logger.info('Notification: %s', notification)

Invoke the service through its REST channel:

curl http://localhost:11223/ward-notification -d '{"family_name": "Smith", "patient_class": "inpatient", "location": "W3, room 12", "event_time": "2026-03-15 12:00"}'

The logged notification reads:

Notification: Smith was admitted as an inpatient to W3, room 12, at 12:00 on 15 March 2026.

The model receives the selected fields rather than the complete admission event. The patient's name and location remain PHI, so the connection determines whether they are sent to a hosted provider or a model operated in the selected environment. The invoke call, its response dictionary and its error behavior are covered under invoking LLMs from services, and conversations that keep their history use chat instead.

Create the connection

In the Dashboard, open AI > LLM connections and create a connection. Services select it through self.llm[name]. Its model setting determines the provider protocol, while the address and API key determine the destination and authentication. Pool size, timeout, token limits and history limits are documented in the connections reference, and the connection can also live as enmasse YAML in git, imported per environment as GitOps describes.

What is recorded and what it costs

Every call through the connection is recorded with its outcome, duration and token usage, for successful and failed calls alike, with the provider's own error kept verbatim, as the audit trail describes. The built-in alerting can measure error rates and latency from those records.

Every response also includes a usage dictionary with input_tokens and output_tokens, using the same two names regardless of the provider:

# Read the usage reported with the response ..
usage = response['usage']

# .. extract the provider-independent token counts ..
input_tokens  = usage['input_tokens']
output_tokens = usage['output_tokens']

# .. and record them.
self.logger.info('Tokens: %s in, %s out', input_tokens, output_tokens)

For more details, the token usage example turns the dictionary into cost per caller, and cost and limits documents the available spending limits.

• Expose services as MCP
• Interface health for AI agents