# Creating LLM connections

Give every service access to Claude, OpenAI, Gemini or self-hosted models - credentials stay in the platform.

An LLM connection is how your Python services call a model - Claude, OpenAI, Gemini or anything self-hosted that implements the OpenAI protocol, such as Ollama, vLLM or LiteLLM. The connection holds the provider's address and [API key](https://zato.io/docs/security/api-keys.html), and services refer to it by name alone, through [`self.llm['My Connection']`](https://zato.io/docs/ai/llm/invoke.html). The key is never visible in service code, and swapping one provider for another is a change to the connection, not to any code.

Connections are managed in the Dashboard under **AI** > **LLM connections**.

## Configuration {#configuration}

| Field | Meaning |
| --- | --- |
| Name | A unique name for this connection - services look it up by this name through `self.llm[name]` |
| Active | Whether this connection can be used - services cannot look up an inactive connection |
| Model | The model every call through this connection uses - pick one from the catalog or enter any name by hand |
| Address | The base URL of the API to call - it follows the model you pick and you can point it at any self-hosted or proxy endpoint |
| API key | The key sent with each call - required for hosted providers, while self-hosted endpoints may not need one |
| Pool size | How many HTTP clients the pool keeps ready - each concurrent call checks one out (default: 50) |
| Timeout | How many seconds to wait for the provider's response (default: 60) |
| Max tokens | The most tokens the model may generate per reply, sent to providers that require it (default: 1024) |
| Max history turns | How many past turns of a [chat](https://zato.io/docs/ai/llm/chat.html) are sent to the provider - a turn is one user message plus the assistant's reply (default: 20) |
| Chat expiry | How many seconds a chat's history is kept after its last message (default: 86400, i.e. one day) |

Once a connection exists, its API key is updated through the **Change API key** link in its row.

## Models and providers {#models-and-providers}

The model you pick determines which protocol the connection uses - there is no separate provider field.

- Names picked from the catalog, such as "Sonnet 5", resolve to their provider and to the id the provider's API expects on the wire.
- Names entered by hand select the protocol by prefix - `claude-` selects the Claude API, `gemini-` the Gemini API, and any other name the OpenAI protocol, which self-hosted and proxied endpoints use as well.

The catalog itself is stored in `default-models.yaml` in the server's `user-conf` directory and the Dashboard reads it afresh each time, so you can add or remove models without a restart.

Each hosted provider has its default address, filled in automatically when you pick its model:

| Provider | Address |
| --- | --- |
| Claude | `https://api.anthropic.com` |
| OpenAI | `https://api.openai.com/v1` |
| Gemini | `https://generativelanguage.googleapis.com/v1beta` |

For a self-hosted model, enter its name by hand and point the address at your endpoint, e.g. `http://localhost:11434/v1` for Ollama. The API key can stay empty when the endpoint does not require one.

## Ping {#ping}

The **Ping** link in a connection's row confirms that the address is reachable and that the key is accepted - it lists the provider's models without generating anything, so a ping costs no tokens.

## The audit trail {#the-audit-trail}

Every call through an LLM connection is recorded in the [audit log](https://zato.io/docs/admin/audit-log/index.html) - the connection's name, the outcome, the duration, the endpoint and the provider's status, `200 OK` or `429 Too Many Requests`, for successful and failed calls alike. Each row is also tagged with four attributes the log can be searched by - `model`, the model the call asked, `finish_reason`, why the provider stopped generating - `stop`, `length`, `refusal` or `tool_use` - and `input_tokens` and `output_tokens`, what the call cost as the provider reported it.

A connection that starts failing, slowing down, running out of tokens or being refused is visible there without any additional configuration.

## See also {#see-also}

- [Invoking LLMs](https://zato.io/docs/ai/llm/invoke.html) - One-shot calls through a connection, with per-call token usage
- [Multi-turn conversations](https://zato.io/docs/ai/llm/chat.html) - Chats whose history the platform keeps and trims
- [Skills](https://zato.io/docs/ai/skills/index.html) - Reusable instructions sent as the system context of a call
- [GitOps](https://zato.io/docs/ai/enmasse.html) - Connections declared in YAML and kept in version control