AI observability

Both directions of AI traffic feed the same platform-wide audit log, built-in alert rules measure rates and latency over those events, and the worst alerts are diagnosed by an LLM before a person looks at them. This page covers the events, the rules and the diagnosis - the storage, the dashboards and the wider monitoring integrations are on their own pages.

The two event streams

The MCP stream is per-gateway opt-in, the LLM stream records every call unconditionally - which is what makes the alert rules below work without any configuration.

The built-in alert rules

The alerting engine periodically sweeps the audit events of each connection and evaluates rules over the window's aggregates - error_rate, error_count, total_count, consecutive_failures and avg_duration_ms. The AI-facing rules that ship with the platform:

All thresholds are defaults of editable rules - the rules screen in the Dashboard is where they change, and the alerting pages describe the rule language, the sweep windows and the notification connections.

The LLM-driven alert diagnosis

Rules whose outcome action is diagnose send the alert to a model before notifying anyone. The diagnosis service assembles an evidence pack - the alert that fired, the connection's configuration with secrets masked, and its recent audit trail, newest first - and sends it to an LLM guided by the built-in diagnostic skill of that connection type, e.g. llm-diagnostics or mcp-diagnostics. The model's diagnosis, its confidence and its remediation advice are stored next to the alert and travel out with its notifications.

The connection that runs the diagnosis is named in the llm_connection key of the rule's action configuration. A rule that names none falls back to the default connection, default.alerts.llm, which ships inactive with placeholder credentials - point it at a real model and activate it, or set the Zato_Alerts_LLM_Connection environment variable to use a different default. With no active connection available, the alert still goes out, without a diagnosis - a person receives the same evidence pack the model would have.

One alert produces one diagnosis - repeated sweeps of the same alert never spend tokens twice.

Trace one request across the layers

Every MCP audit event includes the request's CID - the same correlation ID that appears in the server log and in the audit events of the LLM calls the service made. The tracing one call example follows one request end to end through each layer.

See also

Learn more