Self-hosted models

One service that runs against a hosted provider in production, an Ollama model on a developer's laptop and a corporate LLM gateway in a regulated environment - with zero changes to the service between the three. The switch is made entirely in the connection's configuration, because the service refers only to the connection's name.

Prerequisites. For the self-hosted case, any OpenAI-compatible endpoint - the example uses Ollama with ollama pull llama3.1 already run.

The service

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class Classify(Service):
    """ Classifies a piece of text as positive, negative or neutral.
    """
    name = 'example.ai.classify'

    input = 'text'
    output = 'label'

    def handle(self):

        conn = self.llm['Classifier LLM']

        prompt = 'Answer with one word - positive, negative or neutral: ' + self.request.input.text
        response = conn.invoke(prompt)

        self.response.payload.label = response['text'].strip().lower()

The code contains no address, no key, no model and no protocol - Classifier LLM is the only coupling.

The three connections

The same enmasse entry, three ways. Only one is imported per environment - all three share the name the service looks up:

# Production - a hosted provider
llm:
  - name: Classifier LLM
    model: gpt-4o-mini
    address: https://api.openai.com/v1
    secret: Zato_Enmasse_Env.Classifier_Key

# A laptop - Ollama, without a key
llm:
  - name: Classifier LLM
    model: llama3.1
    address: http://localhost:11434/v1

# A regulated environment - the corporate LLM gateway proxies every call
llm:
  - name: Classifier LLM
    model: company-approved-model
    address: https://llm-gateway.internal.example.com/v1
    secret: Zato_Enmasse_Env.Gateway_Token

No provider field is needed - the model's name selects the protocol, as described under models and providers, and Ollama, vLLM, LiteLLM and corporate gateways all expose the OpenAI protocol.

Run the service

curl localhost:11223/example/classify -d '{"text":"The delivery was fast and the box was intact"}'

Expected output, from any of the three backends:

{"label": "positive"}

The REST channel for the curl call:

channel_rest:
  - name: example.ai.classify
    service: example.ai.classify
    url_path: /example/classify

Verify a backend

The Ping link in the connection's row lists the endpoint's models without generating anything - it confirms the address is reachable and the key is accepted, and a ping costs no tokens, with Ollama and with a hosted provider alike.

Failure behavior

Pointing the address at an endpoint that is down makes invoke raise - with an OpenAI-compatible endpoint that answered with an error, the exception contains the endpoint's own response body, so model "llama3.1" not found from Ollama arrives verbatim instead of as a generic failure. The wrong-address case fails at ping time already, before any service runs. To handle both cases in a service, see provider failures.

See also