# Go-live runbook

Take a clinical interface live - a parallel run, message-count reconciliation and a tested rollback path.

This page takes one clinical interface - an ADT feed, an orders interface, a results interface - from working in a test environment to handling production traffic. The unit of success is a live, reconciled feed: every message the sender emits arrives, is processed and can be accounted for.

Your browser stores the items you have checked off, so you can leave and return to this page at any point. This page assumes the environment itself is production-ready and covers what is specific to a clinical interface.

## Before the cutover {#before-the-cutover}

- [ ] **Confirm the routing with real message samples.** Send representative messages of each type the sender will emit and confirm each reaches the right channel - the [routing rules](https://zato.io/docs/dev/healthcare/hl7v2/mllp/config.html#match-criteria) match on MSH fields, and a message matching no channel is rejected with `AR`.
- [ ] **Decide what happens to the unexpected.** A default channel catches messages no routing rule matches - without one they are rejected.
- [ ] **Match the deduplication window to the sender's retransmission policy.** Clinical systems retransmit when an ACK is late - the [deduplication window](https://zato.io/docs/dev/healthcare/hl7v2/mllp/config.html#deduplication) is what keeps a retransmission from being processed twice.
- [ ] **Serve the traffic over TLS.** Senders connect to port 11554 instead of 11553 and the channels stay unchanged - [connectivity](https://zato.io/docs/dev/healthcare/hl7v2/mllp/index.html#ports-and-tls) covers the ports and [SSL/TLS](https://zato.io/docs/admin/security/ssl/index.html) the certificates.
- [ ] **Turn Return errors off.** With the toggle off - the default - a failing message is answered with a generic `AE` or `AR` and the details stay in your logs, where [error visibility](https://zato.io/docs/dev/healthcare/hl7v2/mllp/acknowledgments.html) belongs in production.
- [ ] **Turn Log messages off.** HL7 messages contain PHI - payload logging is for development, the [audit log](https://zato.io/docs/admin/audit-log/index.html) is the production record.
- [ ] **Export the configuration to git.** The channels, connections and their settings as [enmasse](https://zato.io/docs/admin/enmasse.html) YAML - this is also what makes the rollback below a re-import rather than a reconstruction.

## The parallel run {#the-parallel-run}

Run the new interface alongside the existing path before anything depends on it. The sender feeds both, or feeds a copy of production traffic to the new interface, for an agreed period. This is also how a migration from another engine proceeds, and this runbook is the cutover procedure for each interface being moved.

- [ ] **Watch the feed arrive.** The [audit log](https://zato.io/docs/admin/audit-log/views.html) filtered to the channel shows each message with its acknowledgment, and the activity strip turns the stream into a histogram where a gap or an error burst is immediately visible.
- [ ] **Confirm nothing waits for its other half.** The **Outstanding** filter lists exchanges still missing their acknowledgment, oldest first - at the end of a parallel-run day it should be empty.
- [ ] **Chase every `AE` and `AR` to its cause.** Each failure in the parallel run is one that would have been a production incident - the [troubleshooting page](https://zato.io/docs/dev/healthcare/troubleshooting.html) maps each symptom to its fix.

## Reconciliation {#reconciliation}

- [ ] **Reconcile message counts with the sender.** For each day of the parallel run, the sender's count of messages emitted must equal the audit log's count of messages received - the audit log is filterable by channel and time range.
- [ ] **Spot-check individual messages.** Pick a handful of control IDs from the sender's log and find each in the audit log - the search takes an MSH-10 control ID or a patient identifier and returns the whole exchange it appears in.
- [ ] **Agree on the discrepancy procedure.** Who reconciles, when, and what happens when a count is off by one.

## The cutover {#the-cutover}

- [ ] **Switch the sender to the new interface.** The moment of cutover is a sender-side configuration change - the receiving side is already live and already proven by the parallel run.
- [ ] **Watch the first hour.** The activity strip on the audit log and the sender's operators on the same call - most cutover problems show in the first minutes as either silence or a burst of `AE`.
- [ ] **Reconcile the first day.** The same count reconciliation as the parallel run, once more, against production volume.

## The rollback path {#the-rollback-path}

- [ ] **Know the rollback trigger before you need it.** Agree with the sender's operators what failure justifies rolling back - a reconciliation mismatch, a sustained error rate, silence - and who makes the call.
- [ ] **Rolling back is the cutover, reversed.** The sender points back at the previous path. Nothing on the receiving side needs dismantling - inactive channels have no runtime effect.
- [ ] **No message is lost either way.** Everything received before the rollback is in the audit log with its acknowledgment, and [resubmitting](https://zato.io/docs/admin/audit-log/resubmit.html) can reprocess or resend any of it - including to a single destination that missed its copy.
- [ ] **Configuration is restorable, not reconstructable.** The enmasse YAML in git is the state to return to - a re-import brings the environment back to exactly what was reviewed and tested.

## After go-live {#after-go-live}

The routines that keep a live interface healthy - what queues while a system is down and how the catch-up is protected - are on [downtime and recovery](https://zato.io/docs/dev/healthcare/downtime-and-recovery.html), and the [troubleshooting page](https://zato.io/docs/dev/healthcare/troubleshooting.html) is the symptom-first map for everything that follows.

## Learn more {#learn-more}

- [Healthcare interface engine](https://zato.io/docs/dev/healthcare/) - Clinical messages, FHIR, conversion and operations
- [HL7 v2 parsing](https://zato.io/docs/dev/healthcare/hl7/v2/parsing/) - Messages as typed objects, with field access and validation
- [MLLP channels](https://zato.io/docs/dev/healthcare/hl7v2/mllp/) - Receiving and sending HL7 v2 over MLLP sockets
- [FHIR integrations](https://zato.io/docs/dev/healthcare/hl7/fhir/) - Read and write FHIR resources, with paths, bundles and extensions
- [HL7 v2 to FHIR](https://zato.io/docs/dev/healthcare/hl7/to-fhir/) - Converting v2 messages into FHIR bundles, codes and references included
- [EDIFACT](https://zato.io/docs/dev/healthcare/edifact/) - Parsing interchanges, dialects and the transports they arrive on
- [Transformation](https://zato.io/docs/dev/healthcare/transformation/) - One message in, another standard out, in ordinary Python
- [Audit log](https://zato.io/docs/dev/healthcare/audit-log.html) - Every message with its acknowledgment, searchable by patient identifier
- [AI in clinical interfaces](https://zato.io/docs/dev/healthcare/ai/) - Services calling LLMs and AI agents calling clinical services as tools