# Self-hosted models

Run the same service against hosted, self-hosted and gateway-proxied models without code changes.

One service that runs against a hosted provider in production, an Ollama model on a developer's laptop and a corporate LLM gateway in a regulated environment - with zero changes to the service between the three. The switch is made entirely in the connection's configuration, because the service refers only to the connection's name.

**Prerequisites.** For the self-hosted case, any OpenAI-compatible endpoint - the example uses [Ollama](https://ollama.com) with `ollama pull llama3.1` already run.

## The service {#the-service}

```python
# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class Classify(Service):
    """ Classifies a piece of text as positive, negative or neutral.
    """
    name = 'example.ai.classify'

    input = 'text'
    output = 'label'

    def handle(self):

        conn = self.llm['Classifier LLM']

        prompt = 'Answer with one word - positive, negative or neutral: ' + self.request.input.text
        response = conn.invoke(prompt)

        self.response.payload.label = response['text'].strip().lower()
```

The code contains no address, no key, no model and no protocol - `Classifier LLM` is the only coupling.

## The three connections {#the-three-connections}

The same [enmasse](https://zato.io/docs/ai/enmasse.html) entry, three ways. Only one is imported per environment - all three share the name the service looks up:

```yaml
# Production - a hosted provider
llm:
  - name: Classifier LLM
    model: gpt-4o-mini
    address: https://api.openai.com/v1
    secret: Zato_Enmasse_Env.Classifier_Key
```

```yaml
# A laptop - Ollama, without a key
llm:
  - name: Classifier LLM
    model: llama3.1
    address: http://localhost:11434/v1
```

```yaml
# A regulated environment - the corporate LLM gateway proxies every call
llm:
  - name: Classifier LLM
    model: company-approved-model
    address: https://llm-gateway.internal.example.com/v1
    secret: Zato_Enmasse_Env.Gateway_Token
```

No provider field is needed - the model's name selects the protocol, as described under [models and providers](https://zato.io/docs/ai/llm/connections.html#models-and-providers), and Ollama, vLLM, LiteLLM and corporate gateways all expose the OpenAI protocol.

## Run the service {#run-the-service}

```bash
curl localhost:11223/example/classify -d '{"text":"The delivery was fast and the box was intact"}'
```

Expected output, from any of the three backends:

```json
{"label": "positive"}
```

The REST channel for the `curl` call:

```yaml
channel_rest:
  - name: example.ai.classify
    service: example.ai.classify
    url_path: /example/classify
```

## Verify a backend {#verify-a-backend}

The **Ping** link in the connection's row lists the endpoint's models without generating anything - it confirms the address is reachable and the key is accepted, and a ping costs no tokens, with Ollama and with a hosted provider alike.

## Failure behavior {#failure-behavior}

Pointing the address at an endpoint that is down makes `invoke` raise - with an OpenAI-compatible endpoint that answered with an error, the exception contains the endpoint's own response body, so `model "llama3.1" not found` from Ollama arrives verbatim instead of as a generic failure. The wrong-address case fails at ping time already, before any service runs. To handle both cases in a service, see [provider failures](https://zato.io/docs/ai/examples/provider-failures.html).

## See also {#see-also}

- [LLM connections](https://zato.io/docs/ai/llm/connections.html) - Models, providers, addresses and how the protocol is selected
- [Everything as code](https://zato.io/docs/ai/examples/everything-as-code.html) - The per-environment YAML that swaps the backends
- [Provider failures](https://zato.io/docs/ai/examples/provider-failures.html) - A service that survives its provider going down