# Production best practices

The configuration to apply before a Zato environment receives production traffic, grouped by deployment type and ranked by importance.

> **1. Every environment**
>
> This chapter applies to every production environment, including one that runs on the default configuration.
>
> > **1.1 Password configuration (Every environment, High importance)**
> >
> > Zato provides three built-in user accounts. Administrators sign in to the Dashboard as `admin`, connect to the container over SSH as `zato`, and deploy code from an IDE as `ide_publisher`.
> >
> > | Account | Used to | Environment variable |
> > | --- | --- | --- |
> > | `admin` | Sign in to the Dashboard | `Zato_Dashboard_Password` |
> > | `zato` | Connect to the container over SSH | `Zato_SSH_Password` |
> > | `ide_publisher` | Deploy code from an IDE through [IDE integration](https://zato.io/docs/dev/ide/index.html) | `Zato_IDE_Password` |
> >
> > Zato reads the password of each account from its environment variable when the container starts. If the variable is not set, Zato generates a random password for the account at every start, and the password changes with each restart.
> >
> > Before you start a production container, set each of the three variables to its own password, taken from the secrets store of your organization. Do not use `Zato_Password` in production - it assigns one password to all three accounts and is intended for development environments.
> >
> > ```bash
> > docker run \
> >   -e Zato_Dashboard_Password="$ZATO_DASHBOARD_PASSWORD" \
> >   -e Zato_SSH_Password="$ZATO_SSH_PASSWORD" \
> >   -e Zato_IDE_Password="$ZATO_IDE_PASSWORD" \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > To change a password later, set the new value in the variable and restart the container, as described in [changing passwords](https://zato.io/docs/admin/security/change-password.html).
>
> > **1.2 Dashboard access (Every environment, High importance)**
> >
> > The Dashboard gives full control over the environment, so make it reachable from your internal network only. Publish the Dashboard port on an address of the internal network, and administrators connect to it from the office or through a VPN.
> >
> > ```bash
> > docker run \
> >   -p 10.0.0.5:8184:8184 \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > A Dashboard session lasts two hours by default. To change it, set `Zato_Dashboard_Session_Timeout` to the number of seconds a session lasts.
>
> > **1.3 Development settings (Every environment, High importance)**
> >
> > Leave the following variables unset in production. They are meant for development environments.
> >
> > | Variable | What it does |
> > | --- | --- |
> > | `Zato_Log_Env_Details` | Writes the passwords of the `admin`, `zato` and `ide_publisher` accounts to files inside the container and to `server.log` |
> > | `Zato_Password` | Assigns one password to all three accounts, instead of the [separate passwords](https://zato.io/docs/admin/production-best-practices.html#passwords) |
> > | `Zato_Skip_SSL_Verify` | Stops Zato from verifying the certificates of the systems that it connects to |
> > | `Zato_Dashboard_Debug` | Shows error details, including parts of the Dashboard configuration, in the browser of anyone who causes an error |
> >
> > If you start the production environment from a Compose file or a script copied from a development environment, remove these variables from it.
>
> > **1.4 Image digest and automatic updates (Every environment, Medium importance)**
> >
> > Zato fetches the latest Zato code each time the container starts, so two starts of the same image can run different code. In production, run the code that you tested - refer to the image by its digest instead of `latest`, and set `Zato_Install_Updates` to `False`. With this setting, Zato also stops the automatic updates scheduled in the Dashboard.
> >
> > ```bash
> > docker run \
> >   -e Zato_Install_Updates=False \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > `docker images --digests zatosource/zato-4.1` shows the digest of each image that you pulled. To move to a new version, test the new image and then change the digest, as described in [updating environments](https://zato.io/docs/admin/guide/install/update.html).
>
> > **1.5 Log rotation (Every environment, Low importance)**
> >
> > Zato rotates `server.log` and `http_access.log` when a file reaches 1 GB, and keeps two rotated copies of each. To change it, set `Zato_Server_Log_Max_Size` to the size in bytes and `Zato_Server_Log_Backup_Count` to the number of copies, and restart the container.
> >
> > Docker keeps the container output in a file of its own, which has no size limit by default. Limit it with the `--log-opt` options of `docker run`:
> >
> > ```bash
> > docker run \
> >   -e Zato_Server_Log_Max_Size=500000000 \
> >   -e Zato_Server_Log_Backup_Count=4 \
> >   --log-opt max-size=100m \
> >   --log-opt max-file=3 \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
>
> > **1.6 Environment name (Every environment, Low importance)**
> >
> > Set `Zato_Env_Name` to the name of the environment, for instance `Production`. Zato prints the name in the container output when the environment starts, so `docker logs` shows which environment the output comes from.
> >
> > ```bash
> > docker run \
> >   -e Zato_Env_Name=Production \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```

> **2. SSL**
>
> > **2.1 Configure SSL (SSL, High importance)**
> >
> > Configure SSL/TLS for every environment that API clients or administrators reach over a network, as described in [SSL/TLS configuration](https://zato.io/docs/admin/security/ssl/index.html).
> >
> > Use a certificate issued by the certificate authority of your organization instead of the self-signed certificate that Zato generates, as described in [use your own SSL certificate](https://zato.io/docs/admin/security/ssl/index.html#use-your-own-ssl-certificate).
>
> > **2.2 Published ports (SSL, High importance)**
> >
> > Each Zato interface has a plain-text port and a TLS port. Docker makes a port reachable from outside the host only if you publish it with the `-p` option of `docker run`.
> >
> > | Interface | Used by | Plain-text port | TLS port |
> > | --- | --- | --- | --- |
> > | Dashboard | Administrators | 8183 | 8184 |
> > | API | API clients | 11223 | 11224 |
> > | OpenAPI console | API consumers who read the API documentation | 8185 | 8186 |
> > | HL7 MLLP | Clinical systems | 11553 | 11554 |
> >
> > Publish only the TLS ports of the interfaces you use. Do not publish the plain-text ports, and do not publish port 22, which is used for SSH.
> >
> > ```bash
> > docker run \
> >   -p 8184:8184 \
> >   -p 11224:11224 \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```

> **3. REST and HTTP APIs**
>
> This chapter applies when API clients call the environment over REST or HTTP.
>
> > **3.1 Request size and rate limits (REST and HTTP APIs, Medium importance)**
> >
> > The server accepts HTTP requests of up to 8 MB, headers included, and closes the connection of a larger request without a response. Set `Zato_TCP_Max_Msg_Size`, in whole megabytes, to the size of the largest message that your APIs expect:
> >
> > ```bash
> > docker run \
> >   -e Zato_TCP_Max_Msg_Size=2 \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > Rate limits are off by default. Set them on the channels that callers outside your organization use, or on the security definitions of those callers, so that one caller cannot take up the capacity of the environment. A caller over a limit receives HTTP 429 with a `Retry-After` header. The details are in [rate limiting](https://zato.io/docs/admin/guide/rate-limit.html).

> **4. Outgoing connections**
>
> This chapter applies when services call external systems through outgoing connections.
>
> > **4.1 Outbound network access (Outgoing connections, High importance)**
> >
> > Allow outbound connections from the host only to the systems that Zato needs to reach:
> >
> > - The systems behind your outgoing connections, which health checks call as well
> > - The systems that receive logs or metrics, if Zato sends them out
> > - Your package index, if you install [your own Python packages](https://zato.io/docs/dev/python-packages.html)
> >
> > The [ports](https://zato.io/docs/admin/ports.html) page lists the connections that Zato accepts and makes.
>
> > **4.2 Health checks for outgoing connections (Outgoing connections, High importance)**
> >
> > Turn on health checks for the outgoing REST, SOAP and FHIR connections that your integrations depend on. Zato then pings each connection at a fixed interval, whether or not services use it, so you learn that a system is down before a request needs it. Health checks are off by default.
> >
> > In Dashboard, edit the connection, open the `Alerts` tab and set `Ping every` under `Health check`. In enmasse:
> >
> > ```yaml
> > outgoing_rest:
> >   - name: crm
> >     health_check_run_every: 5
> >     health_check_run_unit: minutes
> > ```
> >
> > Zato records each failed check in the server log and in the audit log. The details are in [health checks](https://zato.io/docs/admin/health-checks.html).

> **5. Scheduled jobs**
>
> This chapter applies when the scheduler runs jobs.
>
> > **5.1 Job schedules and daylight saving time (Scheduled jobs, Low importance)**
> >
> > Jobs repeat at fixed intervals in UTC. A daily job therefore runs at the same UTC time all year, and its local time moves by an hour when daylight saving time begins or ends. No run is skipped or repeated at these changes.
> >
> > If a job must run at the same local time all year, e.g. a report that users expect at 08:00, change its start date when daylight saving time begins and ends. The details are in [scheduler](https://zato.io/docs/dev/examples/scheduler.html).

> **6. Enmasse**
>
> This chapter applies when the configuration of the environment is kept in enmasse files.
>
> > **6.1 Secrets in enmasse files (Enmasse, High importance)**
> >
> > Keep passwords out of enmasse files. Refer to each password through an [environment variable](https://zato.io/docs/admin/enmasse.html#environment-variables), and supply the values from the secrets store of your organization when the container starts.
> >
> > Zato imports the enmasse file each time the container starts, so every start restores the passwords that the file refers to. If you change a password in the Dashboard, change it in the secrets store as well, or the next start restores the previous one.

> **7. Behind a proxy or load balancer**
>
> This chapter applies when a reverse proxy or a load balancer receives requests before Zato does.
>
> > **7.1 Forwarded headers (Behind a proxy or load balancer, High importance)**
> >
> > If a reverse proxy or a load balancer in front of Zato sets `X-Forwarded-For`, set `Zato_Trust_Forwarded_Headers` to `True`. Zato then takes the address of the caller from that header, so IP filtering, rate limits and the access log use the address of the caller instead of the address of the proxy.
> >
> > ```bash
> > docker run \
> >   -e Zato_Trust_Forwarded_Headers=True \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > Leave the variable unset when callers connect to Zato directly. Otherwise, a caller can send `X-Forwarded-For` itself and choose the address that IP filtering and rate limits apply to. The details are in [client IP resolution](https://zato.io/docs/admin/guide/rate-limit.html#requests-forwarded-by-a-load-balancer-or-cdn).
>
> > **7.2 Dashboard trusted origins (Behind a proxy or load balancer, Medium importance)**
> >
> > If administrators open the Dashboard through a proxy, set `Zato_Dashboard_CSRF_Trusted_Origins` to the address that they open in the browser, with the scheme and with the port if it is not the default one, for instance `https://dashboard.example.com`. Separate several addresses with commas. Otherwise, Dashboard forms fail with the error "CSRF verification failed". The details are in [set CSRF trusted origins for the Dashboard](https://zato.io/docs/admin/security/ssl/index.html#set-csrf-trusted-origins-for-the-dashboard).

> **8. Kubernetes and other orchestrators**
>
> This chapter applies when Kubernetes or another orchestrator starts, restarts and replaces the containers.
>
> > **8.1 Liveness and readiness probes (Kubernetes and other orchestrators, High importance)**
> >
> > Point the readiness and liveness probes at [`/zato/ping`](https://zato.io/docs/admin/ping.html) on port 11223. Ping does not call databases or the systems behind outgoing connections, so an outage of those systems does not restart Zato. Do not probe `/`, which answers even when the server is down.
> >
> > ```yaml
> > readinessProbe:
> >   httpGet:
> >     path: /zato/ping
> >     port: 11223
> >   periodSeconds: 10
> >   timeoutSeconds: 5
> >   failureThreshold: 3
> >
> > livenessProbe:
> >   httpGet:
> >     path: /zato/ping
> >     port: 11223
> >   periodSeconds: 30
> >   timeoutSeconds: 5
> >   failureThreshold: 3
> > ```
> >
> > The Helm chart sets up the probes for you. The details are in [health checks](https://zato.io/docs/admin/health-checks.html#using-the-ping-endpoint-as-a-probe).
>
> > **8.2 Start period (Kubernetes and other orchestrators, Medium importance)**
> >
> > Each start builds the whole environment and takes a few minutes. Add a startup probe that allows up to 10 minutes, so that the liveness probe does not restart the container before it is ready. Kubernetes runs the readiness and liveness probes only after the startup probe succeeds.
> >
> > ```yaml
> > startupProbe:
> >   httpGet:
> >     path: /zato/ping
> >     port: 11223
> >   initialDelaySeconds: 30
> >   periodSeconds: 10
> >   timeoutSeconds: 5
> >   failureThreshold: 60
> > ```
> >
> > The details are in [Kubernetes](https://zato.io/docs/admin/guide/kubernetes.html#health-probes).

> **9. External databases**
>
> This chapter applies when the audit log or pub/sub messages must survive the replacement of a container.
>
> > **9.1 External databases for the audit log and pub/sub (External databases, High importance)**
> >
> > The audit log and pub/sub on an external database, with the configuration exported to [enmasse](https://zato.io/docs/admin/enmasse.html) and kept in Git. The SQLite databases inside a container, the [ODB](https://zato.io/docs/admin/odb.html) among them, are deleted when the container is replaced, and the ODB is built again from the enmasse files at the next start.
> >
> > ```text
> > Diagram: what survives a container replacement
> >
> > Column 1 - inside the container, lost on replacement
> >   SQLite ODB, SQLite audit log, SQLite pub/sub, logs without a mount
> >
> > Column 2 - outside the container, retained
> >   Enmasse files in Git, external audit log and pub/sub databases, mounted logs
> >
> > An arrow from an old container to a new one, with column 2 attached to both.
> > ```

> **10. Monitoring and log collection**
>
> This chapter applies when an external system collects metrics or logs from the environment.
>
> > **10.1 Metrics endpoint authentication (Monitoring and log collection, Medium importance)**
> >
> > The `/metrics` endpoint requires HTTP Basic Auth with the user `metrics`. Set `Zato_Metrics_Password` to a password from the secrets store of your organization, and give the same password to the system that collects the metrics. Without the variable, Zato generates a new password at each start, and the collector stops receiving metrics after a restart.
> >
> > ```bash
> > docker run \
> >   -e Zato_Metrics_Password="$ZATO_METRICS_PASSWORD" \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > The details are in [monitoring](https://zato.io/docs/monitoring/index.html).
>
> > **10.2 Log aggregation (Monitoring and log collection, Medium importance)**
> >
> > Zato writes the server log to the container output as well as to `server.log`. The container empties its logs directory at each start, so send the container output to your log platform through a Docker logging driver:
> >
> > ```bash
> > docker run \
> >   --log-driver=fluentd \
> >   --log-opt fluentd-address=logs.example.com:24224 \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > The details are in [logging](https://zato.io/docs/admin/logging.html).
>
> > **10.3 Reading the server log with a log agent (Monitoring and log collection, Medium importance)**
> >
> > A log agent, such as the Datadog Agent, can read the container output, which carries the server log. Each entry in the server log begins with a date, so configure the agent to start a new record at each date - otherwise, every line of a traceback becomes a record of its own.
> >
> > With Datadog, set `DD_LOGS_ENABLED=true` on the Datadog Agent and label the Zato container:
> >
> > ```bash
> > docker run \
> >   -l com.datadoghq.ad.logs='[{"source": "zato", "service": "zato", "log_processing_rules": [{"type": "multi_line", "name": "new_entry", "pattern": "\\d{4}-\\d{2}-\\d{2}"}]}]' \
> >   zatosource/zato-4.1@sha256:<digest>
> > ```
> >
> > The metrics side of Datadog is in [Datadog](https://zato.io/docs/monitoring/datadog.html).

## See also {#see-also}

- [Security defaults](https://zato.io/docs/admin/security/defaults.html) - Generated passwords, inactive channels and the audit log
- [Environment variables](https://zato.io/tutorials/devops/env-variables.html) - Every variable on this page, with its default
- [Updating environments](https://zato.io/docs/admin/guide/install/update.html) - Replacing a container with a newer image

## Learn more {#learn-more}

- [Audit log](https://zato.io/docs/admin/audit-log/) - Every request, response and error, and what you can do with them afterwards
- [Health checks](https://zato.io/docs/admin/health-checks.html) - Have Zato ping your outgoing connections instead of waiting for traffic to fail
- [Logging](https://zato.io/docs/admin/logging.html) - What is written where, and how to ship it to Splunk, Loki, Elastic or Datadog
- [GitOps with enmasse](https://zato.io/docs/admin/enmasse.html) - Keep the whole environment in git as YAML and import it anywhere