Logging

This is what Zato log lines look like:

2026-07-09 10:28:42,955 - INFO - 54728:MainThread - zato.server.base.parallel:21 - Starting Zato 4.1+rev.907497b5
2026-07-09 10:28:50,378 - INFO - 54728:Dummy-7 - customer.cache - 20260709-082850-3727-a1bbe9814a16cac8a - customer.cache:21 - Refreshed the customer cache, entries: 128

The log files

Everything is written to files under the environment directory, which in a container is /opt/zato/env/qs-1/.

Under server1/logs/:

The Dashboard writes under web-admin/logs/ and the scheduler under scheduler/logs/.

Shipping logs to a log platform

Two steps, the same for every platform:

1. Mount the logs directory as a volume, so the files are visible outside the container.
2. Point a collector at the files.

services:
  zato:
    image: zatosource/zato:4.1
    volumes:
      - ./logs:/opt/zato/env/qs-1/server1/logs

Parsing the format

The log format is fixed - plain text, one formatter, and it cannot be switched to JSON. The format string is:

%(asctime)s - %(levelname)s - %(process)d:%(threadName)s -%(zato_ctx)s %(name)s:%(lineno)d - %(message)s

A Grok pattern that handles both shapes, with the optional group covering the service and the CID:

%{TIMESTAMP_ISO8601:timestamp} - %{LOGLEVEL:level} - %{NUMBER:pid}:%{DATA:thread} -( %{DATA:service} - %{DATA:cid} -)? %{DATA:logger}:%{NUMBER:line} - %{GREEDYDATA:message}

Keeping tracebacks together

Only the first line of a traceback has a timestamp, so without a multiline rule each line of it is filed as a separate record. A new record starts at a timestamp, everything else continues the previous one:

^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}

Grafana Loki, with Promtail or Alloy

scrape_configs:
  - job_name: zato
    static_configs:
      - targets: [localhost]
        labels:
          job: zato
          __path__: /var/log/zato/*.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}'
          max_wait_time: 3s
      - regex:
          expression: '^(?P<timestamp>\S+ \S+) - (?P<level>\w+) - (?P<pid>\d+):(?P<thread>[^ ]+) -(?: (?P<service>[^ ]+) - (?P<cid>[^ ]+) -)? (?P<logger>[^:]+):(?P<line>\d+) - (?P<message>.*)$'
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: '2006-01-02 15:04:05,000'

Keep service and cid out of the labels - they are high-cardinality and belong in the line, not in Loki's index.

Elasticsearch, with Fluent Bit

[INPUT]
    Name              tail
    Path              /var/log/zato/*.log
    Tag               zato.*
    Multiline.Parser  zato_multiline
    Refresh_Interval  5

[FILTER]
    Name    parser
    Match   zato.*
    Key_Name log
    Parser  zato

[OUTPUT]
    Name   es
    Match  zato.*
    Host   elasticsearch.example.com
    Port   9200
    Index  zato

With the parsers defined as:

[PARSER]
    Name        zato
    Format      regex
    Regex       ^(?<timestamp>[^ ]+ [^ ]+) - (?<level>\w+) - (?<pid>\d+):(?<thread>[^ ]+) -(?: (?<service>[^ ]+) - (?<cid>[^ ]+) -)? (?<logger>[^:]+):(?<line>\d+) - (?<message>.*)$
    Time_Key    timestamp
    Time_Format %Y-%m-%d %H:%M:%S,%L

[MULTILINE_PARSER]
    Name          zato_multiline
    Type          regex
    Flush_Timeout 3000
    Rule          "start_state"  "/^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}/"  "cont"
    Rule          "cont"         "/^(?!\d{4}-\d{2}-\d{2})/"                      "cont"

Splunk, with the universal forwarder

In inputs.conf:

[monitor:///var/log/zato/*.log]
disabled = false
sourcetype = zato
index = main

In props.conf:

[zato]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 23
EXTRACT-zato = ^(?<timestamp>\S+ \S+) - (?<level>\w+) - (?<pid>\d+):(?<thread>[^ ]+) -(?: (?<service>[^ ]+) - (?<cid>[^ ]+) -)? (?<logger>[^:]+):(?<line>\d+) - (?<message>.*)$

LINE_BREAKER with SHOULD_LINEMERGE = false is what keeps tracebacks in one event.

Datadog, with the Agent

In conf.d/zato.d/conf.yaml:

logs:
  - type: file
    path: /var/log/zato/*.log
    service: zato
    source: python
    log_processing_rules:
      - type: multi_line
        name: zato_log_start
        pattern: \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}

source: python gives you Datadog's built-in Python pipeline, which recognises the level and the tracebacks without further configuration.

Correlating logs with traffic

Every line logged while a request is being handled includes the correlation ID, the same CID the caller got back in the X-Zato-CID response header and the same one on every event in the audit log. Once your collector extracts the cid field, searching for one CID gives you the complete path of a single request across every log line it produced.

Log rotation

server.log and http_access.log are rotated by the server itself:

The remaining logs are rotated daily, or sooner if they pass 100 MB, with 30 compressed copies kept.

Log levels

Log levels are set with environment variables when a container starts.

The levels are the standard ones - DEBUG, INFO, WARN, ERROR and CRITICAL. A per-logger variable always wins over the global Zato_Log_Level, which makes combinations like the one below possible - everything at DEBUG except the REST traffic logger:

export Zato_Log_Level=DEBUG
export Zato_Log_Level_REST=WARN

A few more variables configure other parts of a container:

Changing a level without a restart

Dashboard has a Logging screen where each logger's level can be changed while the environment runs. Saving applies immediately, which is what you want when a problem is happening now and you need DEBUG for the next few minutes.

Audit log

Separately from these logs, the audit log records structured events describing the traffic itself - each request received and each response sent, with the payloads. It is a database rather than a file and it is the basis of alerting.

The audit log has a chapter of its own, covering all the event types, the databases and the SSL/TLS options.

Logging from services

Every service has a self.logger object and everything it logs becomes part of the container's output:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class CustomerCache(Service):

    def handle(self):
        self.logger.info('Refreshed the customer cache, entries: 128')

Each line a service logs automatically records the service's name and the correlation ID (CID) of the request being handled, so a single request can be traced across all the log lines it produced, whichever services were involved:

2026-07-09 10:28:50,378 - INFO - 54728:Dummy-7 - customer.cache - 20260709-082850-3727-a1bbe9814a16cac8a - customer.cache:21 - Refreshed the customer cache, entries: 128

Scheduler job logs

Everything a service logs during a scheduler-initiated run is captured per run - each execution of a job keeps its own set of log entries and they are browsable in the Dashboard, under the job's execution history.

On top of what the service itself logs, the scheduler adds its own entries to each run - when the job started, when it completed and how long it took.

Learn more