Prometheus Metrics

Enabling metrics

proxy-hopper run --config config.yaml --metrics --metrics-port 9090

Or via config:

server:
  metrics: true
  metricsPort: 9090

Metrics are served at http://localhost:9090/metrics.

Metrics reference

Request metrics

Metric	Type	Labels	Description
`proxy_hopper_requests_total`	Counter	`target`, `outcome`, `tag`	Total proxied requests
`proxy_hopper_request_duration_seconds`	Histogram	`target`	Outbound request latency
`proxy_hopper_responses_total`	Counter	`target`, `status_code`, `tag`	Upstream HTTP responses by status code
`proxy_hopper_retries_total`	Counter	`target`	Retry attempts
`proxy_hopper_retry_exhaustions_total`	Counter	`target`	Requests that exhausted all retries
`proxy_hopper_active_connections`	Gauge	—	Open client connections

outcome values: success, rate_limited, server_error, connection_error, no_match. The tag label is set from the X-Proxy-Hopper-Tag request header (empty string if not provided).

Queue metrics

Metric	Type	Labels	Description
`proxy_hopper_queue_depth`	Gauge	`target`	Requests waiting for a free IP
`proxy_hopper_queue_wait_seconds`	Histogram	`target`	Time spent waiting in the queue
`proxy_hopper_queue_expired_total`	Counter	`target`	Requests dropped due to queue timeout

IP pool metrics

Metric	Type	Labels	Description
`proxy_hopper_available_ips`	Gauge	`target`	IPs currently available in pool
`proxy_hopper_quarantined_ips`	Gauge	`target`	IPs currently quarantined
`proxy_hopper_ip_quarantine_events_total`	Counter	`target`, `address`, `provider`, `region`	Quarantine events per IP
`proxy_hopper_ip_failure_count`	Gauge	`target`, `address`, `provider`, `region`	Consecutive failure count per IP

Probe metrics

Metric	Type	Labels	Description
`proxy_hopper_probe_success_total`	Counter	`address`, `provider`, `region`	Successful background probes
`proxy_hopper_probe_failure_total`	Counter	`address`, `provider`, `region`, `reason`	Failed background probes
`proxy_hopper_probe_duration_seconds`	Histogram	`address`, `provider`, `region`	Background probe latency
`proxy_hopper_ip_reachable`	Gauge	`address`, `provider`, `region`	`1` if IP passed last probe, `0` if not

Probe reason values: timeout, proxy_unreachable, connection_error, http_error.

Managed auth metrics

Emitted for targets using managed auth (authManaged: true) — see Token Server for the operational context.

Metric	Type	Labels	Description
`proxy_hopper_auth_token_refreshes_total`	Counter	`target`, `ip`, `status`	Token refresh attempts — `status` is `success` or `failure`
`proxy_hopper_auth_token_refresh_duration_seconds`	Histogram	`target`, `ip`	End-to-end refresh duration
`proxy_hopper_auth_broken_ips_current`	Gauge	`target`	IPs currently in `AUTH_BROKEN` state
`proxy_hopper_auth_server_request_duration_seconds`	Histogram	—	Raw HTTP round-trip to the token server

Admin UI metrics panel

The admin UI’s per-target panel (total requests, success rate, avg latency) is backed by GraphQL’s targetMetrics(name: ...), which pulls from one of two sources — never both at once:

Source	When	Cost
In-process counters	Default — `server.prometheusUrl` unset	A handful of extra `Backend` writes per request (the same `Backend` already used for pool state), fire-and-forget so they never add latency to the response path. Only visible to the admin API when it shares a backend with the proxy — a `memory`-backend deployment with a separate admin process shows an all-zero panel, same as it shows stale everything else.
Prometheus (server-side query)	`server.prometheusUrl` set, e.g. `http://prometheus:9090`	Zero request-path cost — in-process counters are skipped entirely when this is set. The admin API queries Prometheus’s HTTP API itself (the same pattern Grafana’s own backend uses) and returns the aggregated numbers; the browser never talks to Prometheus directly, so this doesn’t add a second, unauthenticated way into your metrics. Requires `server.metrics: true` to actually be enabled and scraped into that Prometheus — this reads the same `proxy_hopper_requests_total` / `proxy_hopper_request_duration_seconds` series documented above.

server:
  prometheusUrl: "http://prometheus:9090"   # or PROXY_HOPPER_PROMETHEUS_URL

This is unrelated to metrics/metricsPort above — those control Proxy Hopper’s own /metrics scrape endpoint (Proxy Hopper as a Prometheus target). prometheusUrl is the opposite direction: Proxy Hopper as a Prometheus client, for one small admin-UI panel, nothing else.

Labels

`tag` label

Set by the X-Proxy-Hopper-Tag request header. Use it to identify which endpoints are consuming IPs or triggering quarantine events:

session.headers["X-Proxy-Hopper-Tag"] = "search-api"

# Which endpoint hits rate limits most?
sum by (tag) (rate(proxy_hopper_requests_total{outcome="rate_limited"}[5m]))

# Response status breakdown by endpoint
sum by (tag, status_code) (rate(proxy_hopper_responses_total[5m]))

`provider` and `region` labels

Come from proxyProviders config — enable per-provider and per-region queries:

# Average probe latency by region
avg by (region) (proxy_hopper_probe_duration_seconds)

# Quarantine rate by provider
rate(proxy_hopper_ip_quarantine_events_total[5m]) by (provider)

# Request success rate by target
rate(proxy_hopper_requests_total{outcome="success"}[5m])
  / rate(proxy_hopper_requests_total[5m])

Kubernetes scraping

See Kubernetes Monitoring for Prometheus Operator ServiceMonitor configuration and pod annotation-based scraping.

Getting Started

Concepts

For Administrators

For Developers

For Contributors

Enabling metrics

Metrics reference

Request metrics

Queue metrics

IP pool metrics

Probe metrics

Managed auth metrics

Admin UI metrics panel

Labels

`tag` label

`provider` and `region` labels

Kubernetes scraping

​Enabling metrics

​Metrics reference

​Request metrics

​Queue metrics

​IP pool metrics

​Probe metrics

​Managed auth metrics

​Admin UI metrics panel

​Labels

​tag label

​provider and region labels

​Kubernetes scraping

Enabling metrics

Metrics reference

Request metrics

Queue metrics

IP pool metrics

Probe metrics

Managed auth metrics

Admin UI metrics panel

Labels

`tag` label

`provider` and `region` labels

Kubernetes scraping