Prometheus metric reference

All Zato metrics follow Prometheus naming conventions:

• zato_ prefix on every metric
• _total suffix on all counters
• _seconds for duration measurements
• _bytes for size measurements

For an overview of the endpoint, architecture, and configuration, see Prometheus metrics.

REST channel metrics

zato_rest_channel_requests_total

Labels:

Trigger: Incremented once per completed REST channel request, after the response is written.

Example PromQL:

sum by (channel_name) (rate(zato_rest_channel_requests_total[5m]))

zato_rest_channel_request_duration_seconds

Labels:

Trigger: Observed once per completed REST channel request with the wall-clock duration from request accept to response write.

Example PromQL:

histogram_quantile(0.99,
  sum by (channel_name, le) (rate(zato_rest_channel_request_duration_seconds_bucket[5m]))
)

zato_rest_channel_request_size_bytes

Labels:

Trigger: Observed once per completed REST channel request with the Content-Length of the inbound request body.

Example PromQL:

histogram_quantile(0.95,
  sum by (channel_name, le) (rate(zato_rest_channel_request_size_bytes_bucket[5m]))
)

zato_rest_channel_response_size_bytes

Labels:

Trigger: Observed once per completed REST channel request with the size of the outbound response body.

Example PromQL:

histogram_quantile(0.95,
  sum by (channel_name, le) (rate(zato_rest_channel_response_size_bytes_bucket[5m]))
)

REST outgoing metrics

zato_rest_outgoing_requests_total

Labels:

Trigger: Incremented once per completed outgoing REST request, after the response is received or an error occurs.

Example PromQL:

sum by (connection_name, error_source) (rate(zato_rest_outgoing_requests_total[5m]))

zato_rest_outgoing_request_duration_seconds

Labels:

Trigger: Observed once per completed outgoing REST request with the wall-clock duration from send to response received.

Example PromQL:

histogram_quantile(0.99,
  sum by (connection_name, le) (rate(zato_rest_outgoing_request_duration_seconds_bucket[5m]))
)

zato_rest_outgoing_request_size_bytes

Labels:

Trigger: Observed once per outgoing REST request with the size of the request body sent.

Example PromQL:

histogram_quantile(0.95,
  sum by (connection_name, le) (rate(zato_rest_outgoing_request_size_bytes_bucket[5m]))
)

zato_rest_outgoing_response_size_bytes

Labels:

Trigger: Observed once per completed outgoing REST request with the size of the response body received.

Example PromQL:

histogram_quantile(0.95,
  sum by (connection_name, le) (rate(zato_rest_outgoing_response_size_bytes_bucket[5m]))
)

Service metrics

zato_service_invocations_total

Labels:

Trigger: Incremented once per service invocation after handle() returns or raises.

Example PromQL:

sum by (service_name) (rate(zato_service_invocations_total{outcome="error"}[5m]))
  /
sum by (service_name) (rate(zato_service_invocations_total[5m]))

zato_service_duration_seconds

Labels:

Trigger: Observed once per service invocation with the wall-clock duration of the handle() method.

Example PromQL:

histogram_quantile(0.95,
  sum by (service_name, le) (rate(zato_service_duration_seconds_bucket[5m]))
)

Pub/sub metrics

zato_pubsub_messages_published_total

Labels:

Trigger: Incremented once per message successfully published to a topic via the Redis Streams backend.

Example PromQL:

sum by (topic_name) (rate(zato_pubsub_messages_published_total[5m]))

zato_pubsub_messages_delivered_total

Labels:

Trigger: Incremented once per message successfully delivered to a subscriber (non-expired, acknowledged).

Example PromQL:

sum by (topic_name) (rate(zato_pubsub_messages_delivered_total[5m]))

Server operational metrics

zato_server_info

Labels (set once at startup):

Trigger: Set once when the server process starts. Does not change during the lifetime of the process.

Example PromQL:

zato_server_info

zato_server_uptime_seconds

Labels: None.

Trigger: Updated to the current elapsed time on every /metrics scrape.

Example PromQL:

zato_server_uptime_seconds

zato_server_requests_in_flight

Labels: None.

Trigger: Incremented when a request enters the HTTP handler, decremented when the response is written. Reflects the instantaneous concurrency at scrape time.

Example PromQL:

zato_server_requests_in_flight

zato_server_config_reloads_total

Labels:

Trigger: Incremented once per hot-deploy configuration event after the reload completes or fails.

Example PromQL:

rate(zato_server_config_reloads_total{result="failure"}[5m])

zato_server_config_last_reload_timestamp_seconds

Labels: None.

Trigger: Set to the current Unix timestamp after each successful hot-deploy configuration reload.

Example PromQL:

time() - zato_server_config_last_reload_timestamp_seconds

This gives you the number of seconds since the last successful reload.

zato_outgoing_health

Labels:

Trigger: Updated periodically by the outgoing health-check subsystem.

Example PromQL:

zato_outgoing_health == 0

Returns all unhealthy outgoing connections.

zato_tls_certificate_expiry_timestamp_seconds

Labels:

Trigger: Updated when the certificate store is loaded or refreshed.

Example PromQL:

(zato_tls_certificate_expiry_timestamp_seconds - time()) / 86400

Returns the number of days until each certificate expires. Alert when this drops below 30.

Scheduler metrics

All scheduler metrics are produced by the Rust scheduler process and merged into the server's /metrics response automatically.

zato_scheduler_jobs_total

Labels: None.

Trigger: Updated on every scheduler tick to reflect the current total number of jobs (active and paused).

Example PromQL:

zato_scheduler_jobs_total

zato_scheduler_jobs_active

Labels: None.

Trigger: Updated on every scheduler tick to reflect the current number of enabled jobs.

Example PromQL:

zato_scheduler_jobs_active

zato_scheduler_jobs_in_flight

Labels: None.

Trigger: Updated on every scheduler tick to reflect jobs that have been dispatched but have not yet completed.

Example PromQL:

zato_scheduler_jobs_in_flight

zato_scheduler_ticks_total

Labels: None.

Trigger: Incremented once per iteration of the scheduler's main loop.

Example PromQL:

rate(zato_scheduler_ticks_total[5m])

zato_scheduler_clock_jumps_total

Labels: None.

Trigger: Incremented when the scheduler detects that the wall clock has jumped forward or backward by more than the expected tick interval. This can happen after VM suspension, NTP corrections, or container migration.

Example PromQL:

increase(zato_scheduler_clock_jumps_total[1h])

zato_scheduler_executions_total

Labels:

Trigger: Incremented once per job execution when the job fires, completes, times out, or is skipped.

Example PromQL:

sum by (job_name) (rate(zato_scheduler_executions_total{outcome="ok"}[5m]))
  /
sum by (job_name) (rate(zato_scheduler_executions_total[5m]))

zato_scheduler_execution_duration_seconds

Labels:

Trigger: Observed when a job execution completes (success or timeout) with the wall-clock duration from fire to completion.

Example PromQL:

histogram_quantile(0.99,
  sum by (job_name, le) (rate(zato_scheduler_execution_duration_seconds_bucket[5m]))
)

zato_scheduler_uptime_seconds

Labels: None.

Trigger: Updated periodically by the scheduler's main loop.

Example PromQL:

zato_scheduler_uptime_seconds