Health checks

The ping endpoints of each component, and proactive checks of outgoing connections.

An outgoing REST or SOAP connection can ping itself on a schedule. The result of each ping is recorded, and if enough of them fail you are alerted.

The Connection down rule fires when a connection's three most recent calls all failed, so it can only fire on a connection somebody is calling. On a connection used once a month, a broken one and a healthy one look identical until the day you need it. A health check gives it a heartbeat to fail.

Outgoing REST and outgoing SOAP connections have health checks. No other connection type has them, and channels have none at all - a channel is called by somebody else, so there is nothing for Zato to ping.

The ping endpoints

Two similar-sounding features point in opposite directions:

A ping endpoint, in this section, is called by your monitoring or your orchestrator and reports that Zato itself is up.
A health check, the rest of this page, is run by Zato, on a schedule, against your outgoing connections, and reports that the systems Zato calls are up.

Each component offers an HTTP ping endpoint - a dedicated address that external monitoring tools can use to discern when the component is operational:

Component	TCP port	URL Path	Method	Expected result
Server	17010	`/zato/ping`	GET	200 OK with a JSON pong
Dashboard	8183	`/`	GET	302 redirect to `/zato`

To ping a server:

$ curl -XGET http://localhost:17010/zato/ping
{"pong":"zato","zato_env":{"result":"ZATO_OK","cid":"7cc657097fa0114017beeed6","details":""}}
$

To ping an instance of Dashboard - it answers with a redirect to its login page:

$ curl -I http://localhost:8183/
HTTP/1.1 302 Found
Location: /zato

All the other ports an environment listens on are in the default ports reference.

Using the ping endpoint as a probe

The server's ping endpoint is what container orchestrators should point their probes at.

In Kubernetes:

readinessProbe:
  httpGet:
    path: /zato/ping
    port: 17010
  initialDelaySeconds: 60
  periodSeconds: 10
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /zato/ping
    port: 17010
  initialDelaySeconds: 120
  periodSeconds: 30
  failureThreshold: 5

In Docker Compose:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:17010/zato/ping"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 120s

Note: Give the start-up enough time in start_period and initialDelaySeconds. A liveness probe that runs out of patience before the container is ready restarts it forever.

How it feeds alerting

A ping goes into the audit log like any other call, under its own source, REST checks or SOAP checks. Alerting reads the audit log, so the ordinary rules pick it up with nothing further to configure.

Checks are measured by the same rules on the same thresholds as the connection's traffic, but counted separately from it, so three failed checks alert exactly as three failed calls would. The message names which one it was, for example REST check failed 3 times.

Check events are kept for 7 days, rather than the 30 days the connection's traffic gets. The audit log retention settings can change this per source.

A connection with its audit log switched off raises no alerts, checks or otherwise. The Audit log checkbox on the connection's Config tab decides whether anything is recorded, and alerting has nothing to measure if it is off.

Turning one on

In Dashboard, go to Connections -> Outgoing -> REST, or Connections -> Outgoing -> SOAP, and edit the connection.
Open the Health check tab.
Set Run every to a number and pick its unit - seconds, minutes, hours or days.
Set Notify on to Failures only or Every result.
Set Deliver to to Service, Pub/sub topic or REST connection, then name the one it should deliver to.
Save. Checks begin immediately, there is no start date to set.

Run every, Deliver to and the callback name have to be given together. Fill in some but not all and saving fails with

Health check options require run-every, callback type and
callback name together

. To remove a health check, clear all of them.

Note: The unit on the create form starts at seconds. Typing 5 and saving without changing the unit gives you a ping every five seconds, not every five minutes.

What it sends

The connection's own ping, to the connection's own address. There is no separate health check URL to configure, and nothing to build on the remote side.

The method is HEAD by default. It is the connection's Ping method on the Config tab, so change it to GET if the remote system answers 405 to a HEAD.
Anything from 400 up is a failure, and so is any exception - a timeout, a refused connection, a TLS problem.
The ping uses the connection's own timeout and its own retry policy, so a connection configured to retry will retry the ping too before calling it a failure.

Note: Path parameters are not filled in for a ping. A connection whose URL path is /api/items/{item_id} is pinged at that literal address, which will usually answer 404 and read as a permanently broken connection.

What the callback receives

Whatever you chose under Deliver to receives one message per ping, or only per failed ping if Notify on is set to failures only:

Key	What it contains
`conn_name`	The connection's name
`conn_type`	`rest_outgoing` or `soap_outgoing`
`is_ok`	`true` or `false`
`response_time_ms`	How long the ping took, in milliseconds
`error`	The error text, or an empty string when the ping succeeded

The callback is where you put your own reaction - open a ticket, flip a flag, notify a team. It is independent of alerting, which reads the audit log rather than the callback, so you can use either, both or neither.

What a failed check does

Nothing beyond recording the result and calling the callback. The connection is not disabled and the next call through it proceeds as before. Two things stop checks running at all:

Disabling a connection also stops its health checks, so an inactive connection is not being watched.
Checks run as scheduler jobs, so a container started with Zato_Start_Scheduler=False runs none. With Notify on set to failures only, that is indistinguishable from everything being fine.

In enmasse

The same five keys live on the connection in enmasse:

outgoing_rest:

  - name: inventory.items
    host: https://inventory.example.com
    url_path: /api/items
    ping_method: HEAD

    health_check_run_every: 5
    health_check_run_unit: minutes
    health_check_notify_on: failures
    health_check_callback_type: service
    health_check_callback_name: demo.input-logger

health_check_run_every is the only one enmasse insists on. Left out, health_check_run_unit defaults to minutes and health_check_notify_on to failures.

Note: If you leave the two callback keys out, the checks run and are recorded, so alerting works, but there is nowhere for the result to be delivered and each ping logs a warning instead. Give both keys or expect nothing to arrive.

Valid values are seconds, minutes, hours or days for the unit, failures or all for notify-on, and service, topic or rest for the callback type.

Learn more

Production checklist What to confirm before an environment takes real traffic

Audit log Every request, response and error, and what you can do with them afterwards

Alerting Rules over the audit log, and the seven ways an alert reaches you

Logging What is written where, and how to ship it to Splunk, Loki, Elastic or Datadog

GitOps with enmasse Keep the whole environment in git as YAML and import it anywhere