# Downtime and recovery

What happens while an interface is down, and how the catch-up is protected from double-processing.

Interfaces go down - maintenance windows, network incidents, a downstream system offline for an upgrade. What matters is what happens to the messages in the meantime and how the recovery avoids the two failure modes: losing a message and processing one twice.

## While you are down {#while-you-are-down}

An MLLP sender does not consider a message delivered until the [acknowledgment](https://zato.io/docs/dev/healthcare/hl7v2/mllp/acknowledgments.html) arrives. While your side is unreachable, the sending system keeps the message and retransmits - queueing on the sender's side is how the protocol itself bridges an outage.

Two things follow from that:

- **The backlog arrives on its own.** When your side is back, senders deliver what they queued - there is no manual step to request it.
- **Retransmissions overlap with what already arrived.** A sender that never received an ACK for a message you did in fact process will send that message again during the catch-up.

Retransmissions of already-processed messages are what [deduplication](https://zato.io/docs/dev/healthcare/hl7v2/mllp/config.html#deduplication) exists for. Each message's MSH-10 control ID is remembered for the configured window - a retransmission of an already-processed message is answered with an immediate `AA` and your service is not invoked again. The window has to outlast the longest outage plus the sender's retransmission policy - the default is 14 days.

## After you are back {#after-you-are-back}

- **Monitor the catch-up.** The [audit log](https://zato.io/docs/admin/audit-log/views.html) filtered to the channel shows the burst - the activity strip makes the backlog visible as a spike, and each message records the acknowledgment it received.
- **Confirm nothing is left waiting.** The **Outstanding** filter lists the exchanges whose acknowledgment has not arrived, oldest first - after a recovery it should drain to empty.
- **Reconcile the outage window.** The sender's count of messages emitted during and after the outage against the audit log's count of messages received - the same [reconciliation](https://zato.io/docs/dev/healthcare/go-live.html) as at go-live, applied to one day.

## When a destination is down {#when-a-destination-is-down}

The direction reverses when a system you send to is offline: your side holds the recorded messages and the destination misses its copies.

Every sent message is stored in the audit log with its full body, not only the metadata - so the recovery is [resubmission](https://zato.io/docs/admin/audit-log/resubmit.html), from the same screen where the failures are listed:

- **Resending** sends the stored payload through the same outgoing connection again - the right action when the failure was on the far side or in between, as it is during an outage.
- **A hop resend** targets one destination of several. One incoming message delivered to three systems, one of which was down, is recovered by sending that one destination its copy - nobody who already has the message gets it twice.
- **The classification column helps with the order of work.** Failures whose errors look like timeouts or refused connections are marked transient - resubmitting them as-is can work. Failures pointing at the message content are marked permanent and need a fix before another attempt.

A message that has already been resubmitted says so instead of offering the action again, which is what prevents the same message from being sent twice during a hurried recovery.

## Learn more {#learn-more}

- [Healthcare interface engine](https://zato.io/docs/dev/healthcare/) - Clinical messages, FHIR, conversion and operations
- [HL7 v2 parsing](https://zato.io/docs/dev/healthcare/hl7/v2/parsing/) - Messages as typed objects, with field access and validation
- [MLLP channels](https://zato.io/docs/dev/healthcare/hl7v2/mllp/) - Receiving and sending HL7 v2 over MLLP sockets
- [FHIR integrations](https://zato.io/docs/dev/healthcare/hl7/fhir/) - Read and write FHIR resources, with paths, bundles and extensions
- [HL7 v2 to FHIR](https://zato.io/docs/dev/healthcare/hl7/to-fhir/) - Converting v2 messages into FHIR bundles, codes and references included
- [EDIFACT](https://zato.io/docs/dev/healthcare/edifact/) - Parsing interchanges, dialects and the transports they arrive on
- [Transformation](https://zato.io/docs/dev/healthcare/transformation/) - One message in, another standard out, in ordinary Python
- [Audit log](https://zato.io/docs/dev/healthcare/audit-log.html) - Every message with its acknowledgment, searchable by patient identifier
- [AI in clinical interfaces](https://zato.io/docs/dev/healthcare/ai/) - Services calling LLMs and AI agents calling clinical services as tools