Monitoring & metrics

Your node exposes two health endpoints and a Prometheus metrics endpoint so you can watch it the way you'd watch any production service. All three live on the private listener, not the public one, so they're never exposed to the open internet along with your inference routes.

info

/healthz, /livez and /metrics bind to server.private_listen (default 127.0.0.1:9091, loopback), separate from the public server.listen port (9090). Point your scrape config and uptime checks at the private port. The listener split keeps /metrics off your public ingress without any extra allowlist — see the endpoint reference. If you run probes or a scraper in another container, override the bind address (e.g. NODE_SERVER_PRIVATE_LISTEN=:9091) rather than moving these onto the public port.

Health — /healthz and /livez

Two endpoints, two different questions. Both report process health only — neither tells you whether your inference backend is reachable or how many models you're serving. For that, watch the metrics below.

Endpoint	Question it answers	On a serving node	While shutting down
GET /healthz	Readiness — should traffic go here?	200 {"status":"ok"}	503 {"status":"draining","inflight_inference":N}
GET /livez	Liveness — is the process wedged?	200 {"status":"ok"}	200 {"status":"draining",…}

The /healthz 503 is deliberate: it's what pulls the node out of your load balancer the moment a graceful shutdown begins, before any caller can be sent to a node that's about to exit. Use it for readiness probes and for uptime monitors that understand "draining" is a normal state.

warning

Don't point a liveness probe at /healthz. It reports unhealthy for the entire drain, so a liveness check on it tells your orchestrator to restart the node in the middle of draining — killing the paid inference the drain exists to finish. Use /livez, which stays 200 throughout. If your node build predates /livez (it returns 404), run no liveness probe at all rather than pointing one at /healthz.

Metrics — /metrics

GET /metrics returns a Prometheus exposition. These are the signals worth scraping and alerting on:

Backend health & catalog

Metric	Type	What it tells you
zs_provider_healthy	Gauge	1 when your text backend is reachable, 0 when it's down (after llm.health_check.failure_threshold failed probes) or not yet probed. While 0, the node advertises zero text models and returns 503 on text reserves. Alert on a sustained 0.
zs_advertised_models	Gauge	How many text models you're actually serving right now — the number clients see. This is the count to dashboard for "am I serving what I think I am."
zs_provider_discovered_models	Gauge	Raw count of models your backend's own listing reported on the last probe. A diagnostic, not a serving count — third-party OpenAI-compatible backends often ship an incomplete list, so this can differ from zs_advertised_models.
zs_image_provider_healthy	Gauge	Image-backend (ComfyUI / Comfy Cloud) twin of zs_provider_healthy. Only present when an image backend is configured, and independent of it — a mixed node can have one backend down while the other serves.
zs_image_advertised_models	Gauge	Image-route models you're currently advertising.

Admission & capacity

Metric	Type	What it tells you
zs_reserve_capacity_slots{model}	Gauge	Per-model concurrent-admission ceiling — the model's max_active_tickets (or the node default). The denominator for utilization.
zs_reserve_inflight_slots{model}	Gauge	Slots currently held by reserved / in-flight tickets. capacity − inflight = free slots; 0 free means the next reserve for that model returns 429 no_capacity.
zs_reserve_capacity_denied_total{model}	Counter	Reserves turned away with 429 no_capacity because the model was at its ceiling — the actionable "we're refusing paying callers" signal. Alert on a sustained rate() > 0. Distinct from the per-IP rate limiter below.
zs_reserve_input_budget_over_total{model,reason,outcome}	Counter	Requests whose measured input exceeded the reserved input_count (see zs.reserve). reason is where it was caught — initial_prompt, tool_loop, or chain. outcome is what happened — rejected (plain request refused zero-cost), tool_cutoff (loop ended early with a final answer), or monitored (enforcement off, served anyway). A sustained rate under monitored is the signal to set enforce_input_budget: true; a sustained tool_cutoff means your tool_headroom_per_iteration is too low for the tools you serve.
zs_provider_upstream_rejection_total{model,code}	Counter	Non-2xx responses from your inference backend, labelled by model and error code — a rising rate points at a backend problem, not the network.
zs_ratelimit_denied_total{bucket}	Counter	Requests dropped by the public-facing rate limiter, by bucket (per-IP / per-account). Distinct from no_capacity.

Signing-account balance

Metric	Type	What it tells you
zs_signing_balance_algos	Gauge	Your hot signing address's ALGO balance from the last successful poll. This address fee-pays every on-chain action (settlement, and ACME / IP-sync updates), so a value trending toward zero means those are about to fail. Alert well above the 1 ALGO boot floor (e.g. < 5). Reads 0 before the first poll — cross-check the timestamp below.
zs_signing_balance_last_success_timestamp_seconds	Gauge	Unix time of the last successful balance poll. If it stops advancing, the balance gauge is stale (algod polling is failing) — alert on staleness so a stuck poller isn't mistaken for a healthy balance.

Keeping this account funded is covered in Staking & economics.

Also exposed

Built-in tool activity (zs_builtin_tool_invocations_total, zs_builtin_tool_errors_total, zs_builtin_tool_duration_seconds) and tool-loop outcomes (zs_tool_loop_finished_total{outcome}, zs_tool_loop_iterations, zs_tool_loop_duration_seconds) are exposed when you enable built-in tools, plus the standard go_* / process_* runtime metrics.

Log lines worth alerting on

Beyond metrics, a handful of log lines are worth wiring into alerts:

• upstream LLM unreachable — your backend stopped answering health probes; the node is now advertising zero models and returning 503 on reserves. Paired with upstream LLM reachable on recovery.
• signing-address ALGO balance is low — the signing balance dropped below your warn_below_algos threshold (default 5). Refill before settlement / ACME / IP-sync transactions start failing. Paired with a recovery line once topped up. signing-address balance poll failed means algod is unreachable and the balance metric is stale.
• reserve input budget exceeded on initial body — a decrypted prompt was larger than what its ticket reserved for it. With enforce_input_budget: true (the default) the request was refused before any upstream call, at zero cost to the payer. Isolated lines are a caller sizing badly; a sustained stream from many payers points at your own config — see Tool loops end early, or reserves are refused.
• tool loop reached reserved input budget — INFO, not an error: a tool loop hit its reserved ceiling and was asked for a final answer instead of another round. The user still got a reply. A steady rate means tool_headroom_per_iteration is undersized for the tools you serve.
• oracle fetch failed — the ALGO/USD price feed is unreachable. Non-fatal: reserves still succeed, but the ALGO-fee-in-USD figure is dropped for that cycle. See Troubleshooting.
• settlement: confirmation failed, will retry — a settle transaction was submitted but not confirmed in time; the driver retries automatically. Occasional lines are normal; a sustained stream means the driver isn't draining (see Troubleshooting).
• settlement: terminal settle error; giving up — a ticket needs manual reconciliation (the contract's inactivity backstop still protects the payer's funds).

What to dashboard

• Liveness — /livez 200 rate. Don't alert on the /healthz 200 rate: it drops to 503 for the whole drain on every restart, so it pages you on routine upgrades. Dashboard /healthz as the rotation signal instead — a node that's been 503 far longer than your drain budget is stuck, not draining.
• Backend up — zs_provider_healthy (and zs_image_provider_healthy if you serve images); alert on sustained 0.
• Admission saturation — per-model utilization zs_reserve_inflight_slots / zs_reserve_capacity_slots (1.0 = full, the next reserve 429s), with the page-worthy edge being rate(zs_reserve_capacity_denied_total[5m]) > 0. Briefly hitting full is healthy use of capacity; sustained denials mean you're turning callers away — raise max_active_tickets (with matching backend capacity / VRAM) or add a node.
• Signing float — zs_signing_balance_algos, alerting above the boot floor, plus staleness on the timestamp gauge.
• Backend errors — rate(zs_provider_upstream_rejection_total[5m]) by code.

If a signal here is red, the troubleshooting guide maps the common symptoms to fixes.