> ## Documentation Index
> Fetch the complete documentation index at: https://proxy-hopper.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Prometheus Metrics

> Full reference for all Proxy Hopper Prometheus metrics.

## Enabling metrics

```bash theme={}
proxy-hopper run --config config.yaml --metrics --metrics-port 9090
```

Or via config:

```yaml theme={}
server:
  metrics: true
  metricsPort: 9090
```

Metrics are served at `http://localhost:9090/metrics`.

## Metrics reference

### Request metrics

| Metric                                  | Type      | Labels                         | Description                            |
| --------------------------------------- | --------- | ------------------------------ | -------------------------------------- |
| `proxy_hopper_requests_total`           | Counter   | `target`, `outcome`, `tag`     | Total proxied requests                 |
| `proxy_hopper_request_duration_seconds` | Histogram | `target`                       | Outbound request latency               |
| `proxy_hopper_responses_total`          | Counter   | `target`, `status_code`, `tag` | Upstream HTTP responses by status code |
| `proxy_hopper_retries_total`            | Counter   | `target`                       | Retry attempts                         |
| `proxy_hopper_retry_exhaustions_total`  | Counter   | `target`                       | Requests that exhausted all retries    |
| `proxy_hopper_active_connections`       | Gauge     | —                              | Open client connections                |

`outcome` values: `success`, `rate_limited`, `server_error`, `connection_error`, `no_match`.

The `tag` label is set from the `X-Proxy-Hopper-Tag` request header (empty string if not provided).

### Queue metrics

| Metric                             | Type      | Labels   | Description                           |
| ---------------------------------- | --------- | -------- | ------------------------------------- |
| `proxy_hopper_queue_depth`         | Gauge     | `target` | Requests waiting for a free IP        |
| `proxy_hopper_queue_wait_seconds`  | Histogram | `target` | Time spent waiting in the queue       |
| `proxy_hopper_queue_expired_total` | Counter   | `target` | Requests dropped due to queue timeout |

### IP pool metrics

| Metric                                    | Type    | Labels                                    | Description                      |
| ----------------------------------------- | ------- | ----------------------------------------- | -------------------------------- |
| `proxy_hopper_available_ips`              | Gauge   | `target`                                  | IPs currently available in pool  |
| `proxy_hopper_quarantined_ips`            | Gauge   | `target`                                  | IPs currently quarantined        |
| `proxy_hopper_ip_quarantine_events_total` | Counter | `target`, `address`, `provider`, `region` | Quarantine events per IP         |
| `proxy_hopper_ip_failure_count`           | Gauge   | `target`, `address`, `provider`, `region` | Consecutive failure count per IP |

### Probe metrics

| Metric                                | Type      | Labels                                    | Description                             |
| ------------------------------------- | --------- | ----------------------------------------- | --------------------------------------- |
| `proxy_hopper_probe_success_total`    | Counter   | `address`, `provider`, `region`           | Successful background probes            |
| `proxy_hopper_probe_failure_total`    | Counter   | `address`, `provider`, `region`, `reason` | Failed background probes                |
| `proxy_hopper_probe_duration_seconds` | Histogram | `address`, `provider`, `region`           | Background probe latency                |
| `proxy_hopper_ip_reachable`           | Gauge     | `address`, `provider`, `region`           | `1` if IP passed last probe, `0` if not |

Probe `reason` values: `timeout`, `proxy_unreachable`, `connection_error`, `http_error`.

### Managed auth metrics

Emitted for targets using [managed auth](/concepts/managed-auth) (`authManaged: true`) — see [Token Server](/admin/token-server/overview#observability) for the operational context.

| Metric                                              | Type      | Labels                   | Description                                                 |
| --------------------------------------------------- | --------- | ------------------------ | ----------------------------------------------------------- |
| `proxy_hopper_auth_token_refreshes_total`           | Counter   | `target`, `ip`, `status` | Token refresh attempts — `status` is `success` or `failure` |
| `proxy_hopper_auth_token_refresh_duration_seconds`  | Histogram | `target`, `ip`           | End-to-end refresh duration                                 |
| `proxy_hopper_auth_broken_ips_current`              | Gauge     | `target`                 | IPs currently in `AUTH_BROKEN` state                        |
| `proxy_hopper_auth_server_request_duration_seconds` | Histogram | —                        | Raw HTTP round-trip to the token server                     |

## Admin UI metrics panel

The admin UI's per-target panel (total requests, success rate, avg latency) is backed by GraphQL's [`targetMetrics(name: ...)`](/admin/admin-server/graphql), which pulls from one of two sources — **never both at once**:

| Source                         | When                                                      | Cost                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| In-process counters            | Default — `server.prometheusUrl` unset                    | A handful of extra `Backend` writes per request (the same `Backend` already used for pool state), fire-and-forget so they never add latency to the response path. Only visible to the admin API when it [shares a backend with the proxy](/admin/admin-server/overview#embedded-vs-separate-process) — a `memory`-backend deployment with a *separate* admin process shows an all-zero panel, same as it shows stale everything else.                                                                                                                        |
| Prometheus (server-side query) | `server.prometheusUrl` set, e.g. `http://prometheus:9090` | Zero request-path cost — in-process counters are skipped entirely when this is set. The admin API queries Prometheus's HTTP API itself (the same pattern Grafana's own backend uses) and returns the aggregated numbers; the browser never talks to Prometheus directly, so this doesn't add a second, unauthenticated way into your metrics. Requires `server.metrics: true` to actually be enabled and scraped into that Prometheus — this reads the same `proxy_hopper_requests_total` / `proxy_hopper_request_duration_seconds` series documented above. |

```yaml theme={}
server:
  prometheusUrl: "http://prometheus:9090"   # or PROXY_HOPPER_PROMETHEUS_URL
```

This is unrelated to `metrics`/`metricsPort` above — those control Proxy Hopper's own `/metrics` *scrape* endpoint (Proxy Hopper as a Prometheus target). `prometheusUrl` is the opposite direction: Proxy Hopper as a Prometheus *client*, for one small admin-UI panel, nothing else.

## Labels

### `tag` label

Set by the `X-Proxy-Hopper-Tag` request header. Use it to identify which endpoints are consuming IPs or triggering quarantine events:

```python theme={}
session.headers["X-Proxy-Hopper-Tag"] = "search-api"
```

```promql theme={}
# Which endpoint hits rate limits most?
sum by (tag) (rate(proxy_hopper_requests_total{outcome="rate_limited"}[5m]))

# Response status breakdown by endpoint
sum by (tag, status_code) (rate(proxy_hopper_responses_total[5m]))
```

### `provider` and `region` labels

Come from `proxyProviders` config — enable per-provider and per-region queries:

```promql theme={}
# Average probe latency by region
avg by (region) (proxy_hopper_probe_duration_seconds)

# Quarantine rate by provider
rate(proxy_hopper_ip_quarantine_events_total[5m]) by (provider)

# Request success rate by target
rate(proxy_hopper_requests_total{outcome="success"}[5m])
  / rate(proxy_hopper_requests_total[5m])
```

## Kubernetes scraping

See [Kubernetes Monitoring](/admin/deployment/kubernetes/monitoring) for Prometheus Operator ServiceMonitor configuration and pod annotation-based scraping.