Go-live runbook

This page takes one clinical interface - an ADT feed, an orders interface, a results interface - from working in a test environment to handling production traffic. The unit of success is a live, reconciled feed: every message the sender emits arrives, is processed and can be accounted for.

Your browser remembers which items you have checked off, so you can leave and return to this page at any point. This page assumes the environment itself is production-ready and covers what is specific to a clinical interface.

Before the cutover

• Confirm the routing with real message samples. Send representative messages of each type the sender will emit and confirm each reaches the right channel - the routing rules match on MSH fields, and a message matching no channel is rejected with AR.
• Decide what happens to the unexpected. A default channel catches messages no routing rule matches - without one they are rejected.
• Match the deduplication window to the sender's retransmission policy. Clinical systems retransmit when an ACK is late - the deduplication window is what keeps a retransmission from being processed twice.
• Serve the traffic over TLS. Senders connect to port 11554 instead of 11553 and the channels stay unchanged - connectivity covers the ports and SSL/TLS the certificates.
• Turn Return errors off. With the toggle off - the default - a failing message is answered with a generic AE and the details stay in your logs, where error visibility belongs in production.
• Turn Log messages off. HL7 messages contain PHI - payload logging is for development, the audit log is the production record.
• Create the alert rules now. An alert on failures and one on silence - a feed that stops arriving is as much an incident as one that fails.
• Export the configuration to git. The channels, connections and their settings as enmasse YAML - this is also what makes the rollback below a re-import rather than a reconstruction.

The parallel run

Run the new interface alongside the existing path before anything depends on it. The sender feeds both, or feeds a copy of production traffic to the new interface, for an agreed period. This is also how a migration from another engine proceeds, and this runbook is the cutover procedure for each interface being moved.

• Watch the feed arrive. The audit log filtered to the channel shows each message with its acknowledgment, and the activity strip turns the stream into a histogram where a gap or an error burst is visible at a glance.
• Confirm nothing waits for its other half. The Outstanding filter lists exchanges still missing their acknowledgment, oldest first - at the end of a parallel-run day it should be empty.
• Chase every AE and AR to its cause. Each failure in the parallel run is one that would have been a production incident - the troubleshooting page maps each symptom to its fix.

Reconciliation

• Reconcile message counts with the sender. For each day of the parallel run, the sender's count of messages emitted must equal the audit log's count of messages received - the audit log is filterable by channel and time range.
• Spot-check individual messages. Pick a handful of control IDs from the sender's log and find each in the audit log - the search takes an MSH-10 control ID or a patient identifier and returns the whole exchange it appears in.
• Agree on the discrepancy procedure. Who reconciles, when, and what happens when a count is off by one.

The cutover

• Switch the sender to the new interface. The moment of cutover is a sender-side configuration change - the receiving side is already live and already proven by the parallel run.
• Watch the first hour. The activity strip on the audit log, the alert rules armed, and the sender's operators on the same call - most cutover problems show in the first minutes as either silence or a burst of AE.
• Reconcile the first day. The same count reconciliation as the parallel run, once more, against production volume.

The rollback path

• Know the rollback trigger before you need it. Agree with the sender's operators what failure justifies rolling back - a reconciliation mismatch, a sustained error rate, silence - and who makes the call.
• Rolling back is the cutover, reversed. The sender points back at the previous path. Nothing on the receiving side needs dismantling - inactive channels have no runtime effect.
• No message is lost either way. Everything received before the rollback is in the audit log with its acknowledgment, and resubmitting can reprocess or resend any of it - including to a single destination that missed its copy.
• Configuration is restorable, not reconstructable. The enmasse YAML in git is the state to return to - a re-import brings the environment back to exactly what was reviewed and tested.

After go-live

The routines that keep a live interface healthy - what queues while a system is down and how the catch-up is protected - are on downtime and recovery, and the troubleshooting page is the symptom-first map for everything that follows.

Learn more