Production best practices

Zato provides three built-in user accounts. Administrators sign in to the Dashboard as admin, connect to the container over SSH as zato, and deploy code from an IDE as ide_publisher.

Zato reads the password of each account from its environment variable when the container starts. If the variable is not set, Zato generates a random password for the account at every start, and the password changes with each restart.

Before you start a production container, set each of the three variables to its own password, taken from the secrets store of your organization. Do not use Zato_Password in production - it assigns one password to all three accounts and is intended for development environments.

docker run \
  -e Zato_Dashboard_Password="$ZATO_DASHBOARD_PASSWORD" \
  -e Zato_SSH_Password="$ZATO_SSH_PASSWORD" \
  -e Zato_IDE_Password="$ZATO_IDE_PASSWORD" \
  zatosource/zato-4.1@sha256:<digest>

To change a password later, set the new value in the variable and restart the container, as described in changing passwords.

The Dashboard gives full control over the environment, so make it reachable from your internal network only. Publish the Dashboard port on an address of the internal network, and administrators connect to it from the office or through a VPN.

docker run \
  -p 10.0.0.5:8184:8184 \
  zatosource/zato-4.1@sha256:<digest>

A Dashboard session lasts two hours by default. To change it, set Zato_Dashboard_Session_Timeout to the number of seconds a session lasts.

Leave the following variables unset in production. They are meant for development environments.

If you start the production environment from a Compose file or a script copied from a development environment, remove these variables from it.

Zato fetches the latest Zato code each time the container starts, so two starts of the same image can run different code. In production, run the code that you tested - refer to the image by its digest instead of latest, and set Zato_Install_Updates to False. With this setting, Zato also stops the automatic updates scheduled in the Dashboard.

docker run \
  -e Zato_Install_Updates=False \
  zatosource/zato-4.1@sha256:<digest>

docker images --digests zatosource/zato-4.1 shows the digest of each image that you pulled. To move to a new version, test the new image and then change the digest, as described in updating environments.

Zato rotates server.log and http_access.log when a file reaches 1 GB, and keeps two rotated copies of each. To change it, set Zato_Server_Log_Max_Size to the size in bytes and Zato_Server_Log_Backup_Count to the number of copies, and restart the container.

Docker keeps the container output in a file of its own, which has no size limit by default. Limit it with the --log-opt options of docker run:

docker run \
  -e Zato_Server_Log_Max_Size=500000000 \
  -e Zato_Server_Log_Backup_Count=4 \
  --log-opt max-size=100m \
  --log-opt max-file=3 \
  zatosource/zato-4.1@sha256:<digest>

Set Zato_Env_Name to the name of the environment, for instance Production. Zato prints the name in the container output when the environment starts, so docker logs shows which environment the output comes from.

docker run \
  -e Zato_Env_Name=Production \
  zatosource/zato-4.1@sha256:<digest>

Configure SSL/TLS for every environment that API clients or administrators reach over a network, as described in SSL/TLS configuration.

Use a certificate issued by the certificate authority of your organization instead of the self-signed certificate that Zato generates, as described in use your own SSL certificate.

Each Zato interface has a plain-text port and a TLS port. Docker makes a port reachable from outside the host only if you publish it with the -p option of docker run.

Publish only the TLS ports of the interfaces you use. Do not publish the plain-text ports, and do not publish port 22, which is used for SSH.

docker run \
  -p 8184:8184 \
  -p 11224:11224 \
  zatosource/zato-4.1@sha256:<digest>

The server accepts HTTP requests of up to 8 MB, headers included, and closes the connection of a larger request without a response. Set Zato_TCP_Max_Msg_Size, in whole megabytes, to the size of the largest message that your APIs expect:

docker run \
  -e Zato_TCP_Max_Msg_Size=2 \
  zatosource/zato-4.1@sha256:<digest>

Rate limits are off by default. Set them on the channels that callers outside your organization use, or on the security definitions of those callers, so that one caller cannot take up the capacity of the environment. A caller over a limit receives HTTP 429 with a Retry-After header. The details are in rate limiting.

Allow outbound connections from the host only to the systems that Zato needs to reach:

• The systems behind your outgoing connections, which health checks call as well
• The systems that receive logs or metrics, if Zato sends them out
• Your package index, if you install your own Python packages

The ports page lists the connections that Zato accepts and makes.

Turn on health checks for the outgoing REST, SOAP and FHIR connections that your integrations depend on. Zato then pings each connection at a fixed interval, whether or not services use it, so you learn that a system is down before a request needs it. Health checks are off by default.

In Dashboard, edit the connection, open the Alerts tab and set Ping every under Health check. In enmasse:

outgoing_rest:
  - name: crm
    health_check_run_every: 5
    health_check_run_unit: minutes

Zato records each failed check in the server log and in the audit log. The details are in health checks.

Jobs repeat at fixed intervals in UTC. A daily job therefore runs at the same UTC time all year, and its local time moves by an hour when daylight saving time begins or ends. No run is skipped or repeated at these changes.

If a job must run at the same local time all year, e.g. a report that users expect at 08:00, change its start date when daylight saving time begins and ends. The details are in scheduler.

Keep passwords out of enmasse files. Refer to each password through an environment variable, and supply the values from the secrets store of your organization when the container starts.

Zato imports the enmasse file each time the container starts, so every start restores the passwords that the file refers to. If you change a password in the Dashboard, change it in the secrets store as well, or the next start restores the previous one.

If a reverse proxy or a load balancer in front of Zato sets X-Forwarded-For, set Zato_Trust_Forwarded_Headers to True. Zato then takes the address of the caller from that header, so IP filtering, rate limits and the access log use the address of the caller instead of the address of the proxy.

docker run \
  -e Zato_Trust_Forwarded_Headers=True \
  zatosource/zato-4.1@sha256:<digest>

Leave the variable unset when callers connect to Zato directly. Otherwise, a caller can send X-Forwarded-For itself and choose the address that IP filtering and rate limits apply to. The details are in client IP resolution.

If administrators open the Dashboard through a proxy, set Zato_Dashboard_CSRF_Trusted_Origins to the address that they open in the browser, with the scheme and with the port if it is not the default one, for instance https://dashboard.example.com. Separate several addresses with commas. Otherwise, Dashboard forms fail with the error "CSRF verification failed". The details are in set CSRF trusted origins for the Dashboard.

Point the readiness and liveness probes at /zato/ping on port 11223. Ping does not call databases or the systems behind outgoing connections, so an outage of those systems does not restart Zato. Do not probe /, which answers even when the server is down.

readinessProbe:
  httpGet:
    path: /zato/ping
    port: 11223
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /zato/ping
    port: 11223
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3

The Helm chart sets up the probes for you. The details are in health checks.

Each start builds the whole environment and takes a few minutes. Add a startup probe that allows up to 10 minutes, so that the liveness probe does not restart the container before it is ready. Kubernetes runs the readiness and liveness probes only after the startup probe succeeds.

startupProbe:
  httpGet:
    path: /zato/ping
    port: 11223
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 60

The details are in Kubernetes.

The audit log and pub/sub on an external database, with the configuration exported to enmasse and kept in Git. The SQLite databases inside a container, the ODB among them, are deleted when the container is replaced, and the ODB is built again from the enmasse files at the next start.

Diagram: what survives a container replacement

Column 1 - inside the container, lost on replacement
  SQLite ODB, SQLite audit log, SQLite pub/sub, logs without a mount

Column 2 - outside the container, retained
  Enmasse files in Git, external audit log and pub/sub databases, mounted logs

An arrow from an old container to a new one, with column 2 attached to both.

The /metrics endpoint requires HTTP Basic Auth with the user metrics. Set Zato_Metrics_Password to a password from the secrets store of your organization, and give the same password to the system that collects the metrics. Without the variable, Zato generates a new password at each start, and the collector stops receiving metrics after a restart.

docker run \
  -e Zato_Metrics_Password="$ZATO_METRICS_PASSWORD" \
  zatosource/zato-4.1@sha256:<digest>

The details are in monitoring.

Zato writes the server log to the container output as well as to server.log. The container empties its logs directory at each start, so send the container output to your log platform through a Docker logging driver:

docker run \
  --log-driver=fluentd \
  --log-opt fluentd-address=logs.example.com:24224 \
  zatosource/zato-4.1@sha256:<digest>

The details are in logging.

A log agent, such as the Datadog Agent, can read the container output, which carries the server log. Each entry in the server log begins with a date, so configure the agent to start a new record at each date - otherwise, every line of a traceback becomes a record of its own.

With Datadog, set DD_LOGS_ENABLED=true on the Datadog Agent and label the Zato container:

docker run \
  -l com.datadoghq.ad.logs='[{"source": "zato", "service": "zato", "log_processing_rules": [{"type": "multi_line", "name": "new_entry", "pattern": "\\d{4}-\\d{2}-\\d{2}"}]}]' \
  zatosource/zato-4.1@sha256:<digest>

The metrics side of Datadog is in Datadog.