# Monitoring with Datadog

The Datadog Agent's OpenMetrics check reads the Prometheus endpoint directly, with no Prometheus server in between.

Datadog reads Zato's [Prometheus endpoint](https://zato.io/docs/monitoring/index.html) directly - the Datadog Agent ships with an OpenMetrics check that scrapes any Prometheus-format endpoint, so no Prometheus server of your own is needed in between.

Before starting, make sure the endpoint itself works - the [Prometheus guide](https://zato.io/docs/monitoring/index.html) shows how to verify it with one `curl` command and where to find its password.

## Configuring the Agent {#configuring-the-agent}

On the host or in the cluster where the Datadog Agent runs, create the OpenMetrics check configuration, e.g. in `/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml`:

```yaml
instances:
  - openmetrics_endpoint: http://zato-server:11223/metrics
    namespace: zato
    auth_type: basic
    username: metrics
    password: your-zato-metrics-password
    metrics:
      - zato_.*
```

Then restart the Agent and confirm the check is running:

```bash
sudo systemctl restart datadog-agent
sudo datadog-agent status
```

The output's checks section should list `openmetrics` with no errors. In a Kubernetes cluster, the same configuration goes into the check's autodiscovery annotations or a ConfigMap, per your cluster's Datadog setup.

Two details:

- `namespace: zato` combined with the metric names already starting with `zato_` means the metrics appear in Datadog as `zato.zato_rest_channel_requests_total` and so on. If you prefer shorter names, remap them with the `metrics` mapping syntax, e.g. `- zato_rest_channel_requests_total: rest.requests`
- Counters arrive in Datadog as `.count` metrics and histograms as `.bucket`, `.sum` and `.count` series, which is Datadog's standard Prometheus translation

## Finding the metrics {#finding-the-metrics}

After a minute or two, the metrics appear in **Metrics** -> **Summary** - type `zato` into the search box:

Every label from the Prometheus endpoint becomes a Datadog tag, so `channel_name`, `service_name`, `status_code`, `error_source`, `topic_name` and `job_name` are all available for filtering and grouping.

## Building a dashboard {#building-a-dashboard}

Create a new dashboard and add widgets grouped by the tags:

- **Request rate** - `zato.zato_rest_channel_requests_total.count`, summed by `channel_name`
- **Error ratio** - the same metric filtered to `status_code IN (4xx, 5xx)`, divided by the total, as a query value widget with thresholds
- **Error source** - the same metric filtered to `error_source NOT IN (none)`, grouped by `error_source`, which separates failures of your services from failures of the external systems they call
- **Latency distribution** - `zato.zato_rest_channel_request_duration_seconds.bucket`, grouped by `le`, or as a percentile if you remap the histogram to a Datadog distribution

The finished screen gives the whole team one place to watch API traffic:

## Monitors {#monitors}

A metric monitor on the error ratio is the natural first alert - trigger when errors exceed 1% of traffic over 5 minutes, grouped by `channel_name` so each channel alerts independently. The `error_source` tag belongs in the alert message, because it answers the first question anyone asks when paged - was it us or the system we call.

## Logs {#logs}

Metrics are one part of a Datadog setup - the [logging chapter](https://zato.io/docs/admin/logging.html) shows how to ship Zato's server and HTTP access logs to Datadog as well, so dashboards and log search are available in one place.

## Learn more {#learn-more}

- [Monitoring overview](https://zato.io/docs/monitoring/) - An overview of metrics, dashboards and tracing
- [Grafana](https://zato.io/docs/monitoring/grafana.html) - Dashboards from the Prometheus endpoint, self-hosted or in Grafana Cloud
- [Health checks](https://zato.io/docs/admin/health-checks.html) - Have Zato ping your outgoing connections instead of waiting for traffic to fail
- [Audit log](https://zato.io/docs/admin/audit-log/) - The record of every message, and what you can do with it afterwards
- [Production checklist](https://zato.io/docs/admin/production-checklist.html) - What to confirm before an environment takes real traffic