Health checks

An outgoing REST, SOAP or FHIR connection can ping itself on a schedule. The result of each ping is recorded in the audit log.

A broken connection can only be noticed when somebody is calling it. On a connection used once a month, a broken one and a healthy one look identical until the day you need it. A health check gives it a heartbeat to fail.

Outgoing REST, outgoing SOAP and outgoing FHIR connections have health checks. No other connection type has them, and channels have none at all - a channel is called by somebody else, so there is nothing for Zato to ping.

The ping endpoints

Two similar-sounding features point in opposite directions:

• A ping endpoint, in this section, is called by your monitoring or your orchestrator and reports that Zato itself is up.
• A health check, the rest of this page, is run by Zato, on a schedule, against your outgoing connections, and reports that the systems Zato calls are up.

Each component offers an HTTP ping endpoint - a dedicated address that external monitoring tools can use to discern when the component is operational:

To ping a server:

$ curl -XGET http://localhost:11223/zato/ping
{"is_ok":true, "cid":"20260921-180542-8418-c1e5d65e18f62730a"}
$

To ping an instance of Dashboard - it answers with a redirect to its login page:

$ curl -I http://localhost:8183/
HTTP/1.1 302 Found
Location: /zato

All the other ports an environment listens on are in the default ports reference.

Using the ping endpoint as a probe

The server's ping endpoint is what container orchestrators should point their probes at.

In Kubernetes:

readinessProbe:
  httpGet:
    path: /zato/ping
    port: 11223
  initialDelaySeconds: 60
  periodSeconds: 10
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /zato/ping
    port: 11223
  initialDelaySeconds: 120
  periodSeconds: 30
  failureThreshold: 5

In Docker Compose:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:11223/zato/ping"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 120s

How it is recorded

A ping goes into the audit log like any other call, under its own source, REST checks, SOAP checks or FHIR checks, and is counted separately from the connection's own traffic.

Check events are kept for 7 days, rather than the 30 days the connection's traffic gets. The audit log retention settings can change this per source.

Turning one on

On an outgoing REST connection:

1. In Dashboard, go to Connections -> Outgoing -> REST and edit the connection.
2. Open the Alerts tab.
3. Under Health check, click the Schedule line and set Ping every to a number and its unit - seconds, minutes, hours or days.
4. Save. Checks begin immediately, there is no start date to set.

On an outgoing SOAP connection:

1. In Dashboard, go to Connections -> Outgoing -> SOAP and edit the connection.
2. Open the Health check tab.
3. Set Run every to a number and pick its unit.
4. Save.

On an outgoing FHIR connection:

1. In Dashboard, go to Connections -> Outgoing -> HL7 -> FHIR and edit the connection.
2. Open the Alerts tab.
3. Under Health check, click the Schedule line and set Ping every to a number and its unit.
4. Save.

To remove a health check, clear the number.

What it sends

The connection's own ping, to the connection's own address. There is no separate health check URL to configure, and nothing to build on the remote side.

A FHIR connection's ping is a GET of the server's CapabilityStatement - the resource every FHIR server serves at [base]/CapabilityStatement - so the points below about the method and the URL path apply to REST and SOAP connections alone.

• The method is HEAD by default. It is the connection's Ping method on the Config tab, so change it to GET if the remote system answers 405 to a HEAD.
• Anything from 400 up is a failure, and so is any exception - a timeout, a refused connection, a TLS problem.
• The ping uses the connection's own timeout and its own retry policy, so a connection configured to retry will retry the ping too before calling it a failure.

What a failed check does

Nothing beyond recording the result. The connection is not disabled and the next call through it proceeds as before. Two things stop checks running at all:

• Disabling a connection also stops its health checks, so an inactive connection is not being watched.
• Checks run as scheduler jobs, so a container started with Zato_Start_Scheduler=False runs none, which is indistinguishable from everything being fine.

In enmasse

The same two keys live on the connection in enmasse:

outgoing_rest:

  - name: inventory.items
    host: https://inventory.example.com
    url_path: /api/items
    ping_method: HEAD

    health_check_run_every: 5
    health_check_run_unit: minutes

health_check_run_every is the only one enmasse insists on. Left out, health_check_run_unit defaults to minutes. An outgoing_soap or an outgoing_fhir connection carries the same two keys.

Valid values for the unit are seconds, minutes, hours or days.

Learn more