# Provider failures

Answer from a primary connection and switch to a secondary one when the primary fails.

Providers time out, reject keys, hit quotas and return 429s - and a service that does not handle it fails silently or hangs. The service below answers from a primary connection and uses a secondary one when the primary fails - the caller receives an answer

or an error within the configured timeouts, and every failed call is still recorded in the audit log.

**Prerequisites.** Two [LLM connections](https://zato.io/docs/ai/llm/connections.html), `Primary LLM` and `Fallback LLM` - two providers, or one provider and a [self-hosted model](https://zato.io/docs/ai/examples/self-hosted-models.html) as the fallback.

## The exceptions {#the-exceptions}

A failed call raises an exception - there are no successful responses that contain an error to check for. The exception states the provider's reason verbatim, and a provider that does not answer raises after the connection's `timeout`, as [error handling](https://zato.io/docs/ai/llm/invoke.html#error-handling) describes.

## The service {#the-service}

```python
# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class ResilientAsk(Service):
    """ Answers from the primary connection, falling back to the secondary one.
    """
    name = 'example.ai.resilient-ask'

    input = 'question'
    output = 'answer', 'served_by'

    def handle(self):

        question = self.request.input.question

        # The primary connection answers under normal circumstances ..
        try:
            response = self.llm['Primary LLM'].invoke(question)
            self.response.payload.served_by = 'primary'

        except Exception as e:

            # .. the exception's text is the provider's own reason - log it,
            # because "provider failed" without the reason is not actionable ..
            self.logger.warning('Primary LLM failed, using the fallback: %s', e)

            # .. and the fallback answers instead. If it fails too, the exception
            # propagates - a caller that gets HTTP 500 and retries later is better
            # off than one that gets a made-up answer.
            response = self.llm['Fallback LLM'].invoke(question)
            self.response.payload.served_by = 'fallback'

        self.response.payload.answer = response['text']
```

## The configuration {#the-configuration}

Short timeouts on both connections - an interactive caller cannot use an answer that took a minute to fail:

```yaml
llm:
  - name: Primary LLM
    model: gpt-4o-mini
    address: https://api.openai.com/v1
    secret: Zato_Enmasse_Env.Primary_Key
    timeout: 15

  - name: Fallback LLM
    model: llama3.1
    address: http://llm-fallback.internal.example.com:11434/v1
    timeout: 15

channel_rest:
  - name: example.ai.resilient-ask
    service: example.ai.resilient-ask
    url_path: /example/resilient-ask
```

## Run the service {#run-the-service}

```bash
curl localhost:11223/example/resilient-ask -d '{"question":"What is an integration platform?"}'
```

Expected output with the primary healthy - `"served_by": "primary"`. To see the failure path without waiting for a real outage, deactivate `Primary LLM` in the Dashboard or break its key - the same call now answers with `"served_by": "fallback"` and the server log shows the provider's refusal:

```text
Primary LLM failed, using the fallback: OpenAI request to `https://api.openai.com/v1/chat/completions` failed with HTTP 401 (Primary LLM)
```

## The audit trail {#the-audit-trail}

The service's `try` and `except` covers the caller's side, and the platform's side needs no code. Every call through either connection, failed calls included, is written to [the audit trail](https://zato.io/docs/ai/llm/connections.html#the-audit-trail), so a primary connection that fails on every call while the fallback serves all traffic is clearly visible there.

## See also {#see-also}

- [Invoking LLMs](https://zato.io/docs/ai/llm/invoke.html) - The exceptions a failed call raises
- [Self-hosted models](https://zato.io/docs/ai/examples/self-hosted-models.html) - A local model as the fallback connection