# Exporting the audit log with OpenTelemetry

Send audit log events to an OpenTelemetry Collector or straight to Splunk, Datadog, Elastic or Grafana Cloud as OTLP log records.

The [audit log](https://zato.io/docs/admin/audit-log/index.html) can send a copy of every event it records to an OpenTelemetry Collector, or directly to a log platform that accepts OTLP, as it is written. Each event becomes one OTLP log record with the event's columns, attributes and, for some sources, its `data` document as record attributes, so the platform you already run - a SIEM, Splunk, Datadog, Elastic, Grafana Cloud - searches and alerts on MCP tool calls, REST traffic and configuration changes alongside everything else it collects.

The export is off until an endpoint is configured, and it never changes what the audit log stores. Events stay in the audit database under the usual [retention](https://zato.io/docs/admin/audit-log/index.html#retention), and the Dashboard keeps showing them, whether or not a copy left.

## Turning the export on {#turning-the-export-on}

All servers and the Dashboard read the same environment variables, in the way the `Zato_Audit_Log_DB_*` variables work. They are read once at start, so a change needs a restart.

| Variable | Default | Description |
| --- | --- | --- |
| `Zato_Audit_Log_Export_Endpoint` | (none) | Where to send the records, e.g. `https://otel.example.com:4318`. The export is off without it |
| `Zato_Audit_Log_Export_Protocol` | `http/protobuf` | The OTLP transport - `http/protobuf` or `grpc` |
| `Zato_Audit_Log_Export_Headers` | (none) | Comma-separated `key=value` pairs sent with every request, e.g. `Authorization=Bearer my.token` |
| `Zato_Audit_Log_Export_SSL_CA_File` | (system store) | Path to the CA certificate the collector's certificate is verified against |
| `Zato_Audit_Log_Export_SSL_Cert_File` | (none) | Path to the client certificate, for mutual TLS |
| `Zato_Audit_Log_Export_SSL_Key_File` | (none) | Path to the client private key, for mutual TLS |
| `Zato_Audit_Log_Export_SSL_Verify` | `true` | Whether to verify the collector's certificate and hostname |
| `Zato_Audit_Log_Export_Compression` | `gzip` | How request bodies are compressed - `gzip` or `none` |
| `Zato_Audit_Log_Export_Timeout_Ms` | `10000` | How long one request to the collector may take |
| `Zato_Audit_Log_Export_Batch_Size` | `64` | How many events are sent in one request when the queue is short |
| `Zato_Audit_Log_Export_Max_Batch_Size` | `1024` | How many events one request may carry when the queue has backed up |
| `Zato_Audit_Log_Export_Flush_Interval_Ms` | `1000` | How long a queued event waits before it is sent regardless of how few there are |
| `Zato_Audit_Log_Export_Queue_Size` | `20000` | How many events wait in memory while the collector is slow or away |
| `Zato_Audit_Log_Export_Sources` | (all) | Comma-separated sources to export, e.g. `mcp,rest-channel,config`. Every source is exported when empty |
| `Zato_Audit_Log_Export_Environment` | (none) | The value of the `deployment.environment.name` resource attribute, e.g. `production` |
| `Zato_Audit_Log_Export_Resource_Attributes` | (none) | Comma-separated `key=value` pairs added to the resource attributes of every record |
| `Zato_Audit_Log_Export_Max_Payload_Size` | `65536` | The size cap of each payload attribute, see [payloads](https://zato.io/docs/admin/audit-log/opentelemetry.html#exporting-payloads) below |

The smallest configuration is one variable, pointing at a collector on the same network:

```bash
export Zato_Audit_Log_Export_Endpoint=http://otel-collector:4318
```

The source names accepted by `Zato_Audit_Log_Export_Sources` are the ones the Dashboard's audit log filters on, e.g. `mcp`, `rest-channel`, `rest-outgoing`, `soap-channel`, `sql-outgoing`, `email-imap`, `scheduler`, `config` and `pubsub`. A name the server does not know keeps the export from starting, and the server log says which name it was.

When the export starts, the server log has one line saying so:

```text
Audit export to http://otel-collector:4318 is on, sources: ['mcp', 'rest-channel']
```

## The two protocols {#the-two-protocols}

Over `http/protobuf`, an endpoint without a path gets `/v1/logs` appended, which is where every collector listens for logs, and an endpoint with a path is used exactly as given, which is what a platform with its own path layout needs. Over `grpc`, the endpoint is a host and port, `https://` means a TLS channel and `http://` a plain one, and the headers are sent as call metadata.

```bash
# HTTP - becomes http://otel-collector:4318/v1/logs
export Zato_Audit_Log_Export_Endpoint=http://otel-collector:4318

# gRPC
export Zato_Audit_Log_Export_Endpoint=https://otel-collector:4317
export Zato_Audit_Log_Export_Protocol=grpc
```

## TLS and headers {#tls-and-headers}

An `https` endpoint is verified against the system's CA store unless `Zato_Audit_Log_Export_SSL_CA_File` names a different one, and the certificate and key files turn mutual TLS on. The headers carry whatever the receiving side uses to tell callers apart, and their values are never written to a log line.

The collector of the examples below is yours to run, but the same four variables also point the servers straight at a hosted platform:

```bash
# Splunk - the Splunk Distribution of the OpenTelemetry Collector listens on 4318
# and forwards to Splunk Enterprise or Splunk Cloud over HEC
export Zato_Audit_Log_Export_Endpoint=http://splunk-otel-collector:4318

# Datadog - the Datadog Agent accepts OTLP on 4318 when its OTLP receiver is on
export Zato_Audit_Log_Export_Endpoint=http://datadog-agent:4318

# Elastic - the managed OTLP endpoint of an Elastic Cloud deployment
export Zato_Audit_Log_Export_Endpoint=https://my-deployment.ingest.eu-west-1.aws.elastic.cloud
export Zato_Audit_Log_Export_Headers="Authorization=ApiKey my.api.key"

# Grafana Cloud - the OTLP gateway, with the full logs path and the instance ID and token as Basic auth
export Zato_Audit_Log_Export_Endpoint=https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs
export Zato_Audit_Log_Export_Headers="Authorization=Basic MTIzNDU2OmdsY19leUpyZXhhbXBsZQ=="
```

## A minimal collector {#a-minimal-collector}

This configuration of the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) receives the records over both protocols and writes them to its own output, which is enough to see what arrives before pointing the collector's exporter at your platform:

```yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]
```

## What a record looks like {#what-a-record-looks-like}

Each event is one log record. This is a `tools/call` on an MCP gateway called `billing` by a person who signed in through the customer's identity provider:

```text
timestamp                           2026-09-14T10:28:42.955Z
observed_timestamp                  2026-09-14T10:28:42.961Z
severity_text                       INFO
event_name                          zato.audit.mcp-tools-call
body                                mcp mcp-tools-call billing ok
zato.audit.event_id                 48213
zato.audit.source                   mcp
zato.audit.event_type               mcp-tools-call
zato.audit.object_name              billing
zato.audit.cid                      20260914-102842-3727-a1bbe9814a16cac8a
zato.audit.endpoint                 billing.invoice.get
zato.audit.ext_client_id            mcp.billing.agents/maria.johnson@example.com
zato.audit.sub_key                  e1a0b6d2-4e1f-4a6b-9f6d-5f4e1b2c3d4e
zato.audit.size                     1841
zato.audit.outcome                  ok
zato.audit.attr.identity            maria.johnson@example.com
zato.audit.attr.client              cursor
zato.audit.data.method              tools/call
zato.audit.data.duration_ms         184
zato.audit.data.remote_address      203.0.113.7
zato.audit.data.auth.type           bearer_jwt
zato.audit.data.auth.definition     mcp.billing.agents
zato.audit.data.auth.identity       maria.johnson@example.com
zato.audit.data.auth.issuer         https://login.example.com/realms/corp
zato.audit.data.auth.audience       api://zato-mcp
zato.audit.data.auth.client         cursor
zato.audit.data.auth.scopes         ["api://zato-mcp/tools.access"]
zato.audit.data.pii_removed.email   2
```

The `timestamp` is when the event happened and `observed_timestamp` is when the server handed it to the export. The `event_name` is `zato.audit.` followed by the event type, and the body is the one line `<source> <event type> <object name> <outcome>`, which is what a log viewer shows as the message.

### Columns {#columns}

Every column of the audit log's `event` table except `data` and `server_name` becomes an attribute called `zato.audit.<column>` - `source`, `event_type`, `object_name`, `cid`, `cid_sequence`, `msg_id`, `correl_id`, `ext_client_id`, `endpoint`, `sub_key`, `size`, `priority`, `outcome`, `status`, `application_outcome`, `classification`, `duration_ms` and `pub_time_iso` - and the event's database ID is `zato.audit.event_id`. A column that is an empty string is left out of the record, numbers are always sent, and `server_name` is a resource attribute instead, see below.

### Attributes {#attributes}

The attributes of an event, the values extracted from it that the Dashboard's free-text search finds events by, become `zato.audit.attr.<name>` and keep their type - an integer stays an integer, a boolean stays a boolean.

### The data document {#the-data-document}

The `data` document of an event is flattened under `zato.audit.data.` with nested keys joined by a dot, so the `identity` inside an MCP event's `auth` block arrives as `zato.audit.data.auth.identity`, and a count under `pii_removed` arrives as `zato.audit.data.pii_removed.email`. The prefix keeps `zato.audit.data.duration_ms` apart from the column `zato.audit.duration_ms` when both exist. A list of values of one type stays a list, as `auth.scopes` does above, a list of mixed values becomes a list of strings, and a list holding objects is sent as one JSON string. Nulls and empty strings are left out, and a string longer than 4,096 characters is cut to that length.

Which sources send their `data` depends on what the document holds. Where it is metadata, it leaves in full, and where it is the payload of a message, it stays home unless the object's [payload flag](https://zato.io/docs/admin/audit-log/opentelemetry.html#exporting-payloads) is on:

| Source | What leaves under `zato.audit.data.` |
| --- | --- |
| `mcp` | Every key - the method, the duration, the remote address, the `auth` block and the response control trace |
| `config` | Every key - who changed what, with the before and after summary and secrets masked |
| `file-outgoing` | Every key - the run summary of a file transfer |
| `sql-outgoing` | `statement`, `row_count` and `error` |
| `fhir` | `resource_type`, `method` and `path` |
| `email-imap` | `sent_from` and `sent_to` |
| `as2` | `mic` and `disposition` |
| any source, `alert-raised` events | `kind`, `message`, `rule` and `count` |
| every other source | Nothing, unless the object's payload flag is on |

### Severity {#severity}

The severity of a record follows the event's outcome:

| Outcome | Severity |
| --- | --- |
| `ok`, `running` or empty | `INFO` |
| `expired`, a scheduler job skipped because the previous run was still in flight | `WARN` |
| `error`, a scheduler job that timed out | `ERROR` |

### Resource attributes {#resource-attributes}

The resource of every record says which process sent it. A server sets `service.name` to `zato-server`, `service.namespace` and `zato.cluster.name` to the cluster's name, `service.instance.id` to a value unique to that server's start, `zato.server.name` to the server's name, plus `service.version`, `host.name` and `process.pid`. The Dashboard sends `zato-dashboard` as `service.name` and `dashboard` as `zato.server.name`. `Zato_Audit_Log_Export_Environment` adds `deployment.environment.name` and `Zato_Audit_Log_Export_Resource_Attributes` adds whatever else your platform groups by:

```bash
export Zato_Audit_Log_Export_Environment=production
export Zato_Audit_Log_Export_Resource_Attributes="team=payments,region=eu-west-1"
```

The resource is built from these variables alone. The `OTEL_*` variables that other tooling on the same host may read, such as `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_RESOURCE_ATTRIBUTES`, are not consulted, so they never redirect or relabel the audit stream.

## Exporting payloads {#exporting-payloads}

By default, the bodies of the messages the audit log stores never leave - a REST channel's request and response, a SOAP envelope, an email, a published message, the rows an SQL statement returned. An object whose enmasse definition sets `is_audit_export_payload_active` to `true` sends them too:

```yaml
channel_rest:
  - name: payments.callback
    service: payments.handle-callback
    url_path: /payments/callback
    is_audit_export_payload_active: true
```

The key exists on REST and SOAP channels, outgoing REST and SOAP connections, IMAP and SMTP connections, pub/sub topics, AS2 and AS4, outgoing SQL connections, Kafka channels and connections, FHIR connections and MLLP channels and connections. MCP gateways do not have it, since their `data` is metadata and leaves in full regardless. The flag has no effect on an object whose audit log is off, because nothing is recorded to export then.

With the flag on, a payload-style `data` document goes as `zato.audit.payload.data`, and each stored body goes as `zato.audit.payload.<kind>`, e.g. `zato.audit.payload.request`, `zato.audit.payload.response` or `zato.audit.payload.sql-rows`. Attachments never leave. Each payload attribute is cut at `Zato_Audit_Log_Export_Max_Payload_Size`, 65,536 bytes by default, and a record that was cut carries `zato.audit.payload.truncated` set to `true`.

Three things follow from turning the flag on. A payload that reached the collector is outside the audit log's [retention](https://zato.io/docs/admin/audit-log/index.html#retention), so deleting it from Zato after 30 days does not delete it from the platform that received it. The Dashboard records who viewed a payload in the audit log, and a payload read from the platform leaves no such record. And a queued event with payloads takes as much memory as the payloads it carries, so with the flag on across busy objects the queue of `Zato_Audit_Log_Export_Queue_Size` events can hold many times the 40 MB that 20,000 events without payloads take.

## While the collector is unreachable {#while-the-collector-is-unreachable}

Request handling never waits on the export. An event is appended to an in-memory queue as it is written to the database, and a separate worker sends the queue to the collector in batches. A batch leaves the queue only after the collector accepted it.

When the collector refuses a batch or cannot be reached, the batch stays at the head of the queue and the worker retries it with a wait that doubles from one second to one minute. The queue keeps accepting new events up to `Zato_Audit_Log_Export_Queue_Size`, and once it is full each new event pushes the oldest one out, so what the collector receives after the outage is contiguous up to the present. An outage writes three lines to the [server log](https://zato.io/docs/admin/logging.html), each once:

```text
Audit export to http://otel-collector:4318 paused - HTTP 503 Service Unavailable
Audit export queue full at 20,000 events, dropping the oldest
Audit export to http://otel-collector:4318 resumed, 1,234 events dropped, ids 5,001 to 6,234
```

The second line appears only when the queue filled up, and the third names the dropped events when there were any, so you know exactly which range to read from the audit database, where they still are, under the usual retention and in its archive.

When a server stops, it sends what is still queued for at most `Zato_Audit_Log_Export_Timeout_Ms` and then closes the connection. A collector that is away at that moment loses what was queued, and the events are in the database as always.

## Learn more {#learn-more}

- [Production checklist](https://zato.io/docs/admin/production-checklist.html) - What to confirm before an environment takes real traffic
- [Audit log](https://zato.io/docs/admin/audit-log/) - Every request, response and error, and what you can do with them afterwards
- [Health checks](https://zato.io/docs/admin/health-checks.html) - Have Zato ping your outgoing connections instead of waiting for traffic to fail
- [Logging](https://zato.io/docs/admin/logging.html) - What is written where, and how to ship it to Splunk, Loki, Elastic or Datadog
- [GitOps with enmasse](https://zato.io/docs/admin/enmasse.html) - Keep the whole environment in git as YAML and import it anywhere