Exporting the audit log with OpenTelemetry

The audit log can send a copy of every event it records to an OpenTelemetry Collector, or directly to a log platform that accepts OTLP, as it is written. Each event becomes one OTLP log record with the event's columns, attributes and, for some sources, its data document as record attributes, so the platform you already run - a SIEM, Splunk, Datadog, Elastic, Grafana Cloud - searches and alerts on MCP tool calls, REST traffic and configuration changes alongside everything else it collects.

The export is off until an endpoint is configured, and it never changes what the audit log stores. Events stay in the audit database under the usual retention, and the Dashboard keeps showing them, whether or not a copy left.

Turning the export on

All servers and the Dashboard read the same environment variables, in the way the Zato_Audit_Log_DB_* variables work. They are read once at start, so a change needs a restart.

The smallest configuration is one variable, pointing at a collector on the same network:

export Zato_Audit_Log_Export_Endpoint=http://otel-collector:4318

The source names accepted by Zato_Audit_Log_Export_Sources are the ones the Dashboard's audit log filters on, e.g. mcp, rest-channel, rest-outgoing, soap-channel, sql-outgoing, email-imap, scheduler, config and pubsub. A name the server does not know keeps the export from starting, and the server log says which name it was.

When the export starts, the server log has one line saying so:

Audit export to http://otel-collector:4318 is on, sources: ['mcp', 'rest-channel']

The two protocols

Over http/protobuf, an endpoint without a path gets /v1/logs appended, which is where every collector listens for logs, and an endpoint with a path is used exactly as given, which is what a platform with its own path layout needs. Over grpc, the endpoint is a host and port, https:// means a TLS channel and http:// a plain one, and the headers are sent as call metadata.

# HTTP - becomes http://otel-collector:4318/v1/logs
export Zato_Audit_Log_Export_Endpoint=http://otel-collector:4318

# gRPC
export Zato_Audit_Log_Export_Endpoint=https://otel-collector:4317
export Zato_Audit_Log_Export_Protocol=grpc

TLS and headers

An https endpoint is verified against the system's CA store unless Zato_Audit_Log_Export_SSL_CA_File names a different one, and the certificate and key files turn mutual TLS on. The headers carry whatever the receiving side uses to tell callers apart, and their values are never written to a log line.

The collector of the examples below is yours to run, but the same four variables also point the servers straight at a hosted platform:

# Splunk - the Splunk Distribution of the OpenTelemetry Collector listens on 4318
# and forwards to Splunk Enterprise or Splunk Cloud over HEC
export Zato_Audit_Log_Export_Endpoint=http://splunk-otel-collector:4318

# Datadog - the Datadog Agent accepts OTLP on 4318 when its OTLP receiver is on
export Zato_Audit_Log_Export_Endpoint=http://datadog-agent:4318

# Elastic - the managed OTLP endpoint of an Elastic Cloud deployment
export Zato_Audit_Log_Export_Endpoint=https://my-deployment.ingest.eu-west-1.aws.elastic.cloud
export Zato_Audit_Log_Export_Headers="Authorization=ApiKey my.api.key"

# Grafana Cloud - the OTLP gateway, with the full logs path and the instance ID and token as Basic auth
export Zato_Audit_Log_Export_Endpoint=https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs
export Zato_Audit_Log_Export_Headers="Authorization=Basic MTIzNDU2OmdsY19leUpyZXhhbXBsZQ=="

A minimal collector

This configuration of the OpenTelemetry Collector receives the records over both protocols and writes them to its own output, which is enough to see what arrives before pointing the collector's exporter at your platform:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]

What a record looks like

Each event is one log record. This is a tools/call on an MCP gateway called billing by a person who signed in through the customer's identity provider:

timestamp                           2026-09-14T10:28:42.955Z
observed_timestamp                  2026-09-14T10:28:42.961Z
severity_text                       INFO
event_name                          zato.audit.mcp-tools-call
body                                mcp mcp-tools-call billing ok
zato.audit.event_id                 48213
zato.audit.source                   mcp
zato.audit.event_type               mcp-tools-call
zato.audit.object_name              billing
zato.audit.cid                      20260914-102842-3727-a1bbe9814a16cac8a
zato.audit.endpoint                 billing.invoice.get
zato.audit.ext_client_id            mcp.billing.agents/maria.johnson@example.com
zato.audit.sub_key                  e1a0b6d2-4e1f-4a6b-9f6d-5f4e1b2c3d4e
zato.audit.size                     1841
zato.audit.outcome                  ok
zato.audit.attr.identity            maria.johnson@example.com
zato.audit.attr.client              cursor
zato.audit.data.method              tools/call
zato.audit.data.duration_ms         184
zato.audit.data.remote_address      203.0.113.7
zato.audit.data.auth.type           bearer_jwt
zato.audit.data.auth.definition     mcp.billing.agents
zato.audit.data.auth.identity       maria.johnson@example.com
zato.audit.data.auth.issuer         https://login.example.com/realms/corp
zato.audit.data.auth.audience       api://zato-mcp
zato.audit.data.auth.client         cursor
zato.audit.data.auth.scopes         ["api://zato-mcp/tools.access"]
zato.audit.data.pii_removed.email   2

The timestamp is when the event happened and observed_timestamp is when the server handed it to the export. The event_name is zato.audit. followed by the event type, and the body is the one line <source> <event type> <object name> <outcome>, which is what a log viewer shows as the message.

Columns

Every column of the audit log's event table except data and server_name becomes an attribute called zato.audit.<column> - source, event_type, object_name, cid, cid_sequence, msg_id, correl_id, ext_client_id, endpoint, sub_key, size, priority, outcome, status, application_outcome, classification, duration_ms and pub_time_iso - and the event's database ID is zato.audit.event_id. A column that is an empty string is left out of the record, numbers are always sent, and server_name is a resource attribute instead, see below.

Attributes

The attributes of an event, the values extracted from it that the Dashboard's free-text search finds events by, become zato.audit.attr.<name> and keep their type - an integer stays an integer, a boolean stays a boolean.

The data document

The data document of an event is flattened under zato.audit.data. with nested keys joined by a dot, so the identity inside an MCP event's auth block arrives as zato.audit.data.auth.identity, and a count under pii_removed arrives as zato.audit.data.pii_removed.email. The prefix keeps zato.audit.data.duration_ms apart from the column zato.audit.duration_ms when both exist. A list of values of one type stays a list, as auth.scopes does above, a list of mixed values becomes a list of strings, and a list holding objects is sent as one JSON string. Nulls and empty strings are left out, and a string longer than 4,096 characters is cut to that length.

Which sources send their data depends on what the document holds. Where it is metadata, it leaves in full, and where it is the payload of a message, it stays home unless the object's payload flag is on:

Severity

The severity of a record follows the event's outcome:

Resource attributes

The resource of every record says which process sent it. A server sets service.name to zato-server, service.namespace and zato.cluster.name to the cluster's name, service.instance.id to a value unique to that server's start, zato.server.name to the server's name, plus service.version, host.name and process.pid. The Dashboard sends zato-dashboard as service.name and dashboard as zato.server.name. Zato_Audit_Log_Export_Environment adds deployment.environment.name and Zato_Audit_Log_Export_Resource_Attributes adds whatever else your platform groups by:

export Zato_Audit_Log_Export_Environment=production
export Zato_Audit_Log_Export_Resource_Attributes="team=payments,region=eu-west-1"

The resource is built from these variables alone. The OTEL_* variables that other tooling on the same host may read, such as OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_RESOURCE_ATTRIBUTES, are not consulted, so they never redirect or relabel the audit stream.

Exporting payloads

By default, the bodies of the messages the audit log stores never leave - a REST channel's request and response, a SOAP envelope, an email, a published message, the rows an SQL statement returned. An object whose enmasse definition sets is_audit_export_payload_active to true sends them too:

channel_rest:
  - name: payments.callback
    service: payments.handle-callback
    url_path: /payments/callback
    is_audit_export_payload_active: true

The key exists on REST and SOAP channels, outgoing REST and SOAP connections, IMAP and SMTP connections, pub/sub topics, AS2 and AS4, outgoing SQL connections, Kafka channels and connections, FHIR connections and MLLP channels and connections. MCP gateways do not have it, since their data is metadata and leaves in full regardless. The flag has no effect on an object whose audit log is off, because nothing is recorded to export then.

With the flag on, a payload-style data document goes as zato.audit.payload.data, and each stored body goes as zato.audit.payload.<kind>, e.g. zato.audit.payload.request, zato.audit.payload.response or zato.audit.payload.sql-rows. Attachments never leave. Each payload attribute is cut at Zato_Audit_Log_Export_Max_Payload_Size, 65,536 bytes by default, and a record that was cut carries zato.audit.payload.truncated set to true.

Three things follow from turning the flag on. A payload that reached the collector is outside the audit log's retention, so deleting it from Zato after 30 days does not delete it from the platform that received it. The Dashboard records who viewed a payload in the audit log, and a payload read from the platform leaves no such record. And a queued event with payloads takes as much memory as the payloads it carries, so with the flag on across busy objects the queue of Zato_Audit_Log_Export_Queue_Size events can hold many times the 40 MB that 20,000 events without payloads take.

While the collector is unreachable

Request handling never waits on the export. An event is appended to an in-memory queue as it is written to the database, and a separate worker sends the queue to the collector in batches. A batch leaves the queue only after the collector accepted it.

When the collector refuses a batch or cannot be reached, the batch stays at the head of the queue and the worker retries it with a wait that doubles from one second to one minute. The queue keeps accepting new events up to Zato_Audit_Log_Export_Queue_Size, and once it is full each new event pushes the oldest one out, so what the collector receives after the outage is contiguous up to the present. An outage writes three lines to the server log, each once:

Audit export to http://otel-collector:4318 paused - HTTP 503 Service Unavailable
Audit export queue full at 20,000 events, dropping the oldest
Audit export to http://otel-collector:4318 resumed, 1,234 events dropped, ids 5,001 to 6,234

The second line appears only when the queue filled up, and the third names the dropped events when there were any, so you know exactly which range to read from the audit database, where they still are, under the usual retention and in its archive.

When a server stops, it sends what is still queued for at most Zato_Audit_Log_Export_Timeout_Ms and then closes the connection. A collector that is away at that moment loses what was queued, and the events are in the database as always.

Learn more