Troubleshooting

The symptoms below cover the problems operators hit most often, from first bring-up through day-to-day operation. Each names what you'll see, why it happens, and what to change. If you have metrics scraped, they'll usually confirm which case you're in.

info

Before working through these by hand, run zs-node doctor. It checks your config against your live backend and the chain in one pass, and it catches the whole class of problem where your config advertises something your backend won't actually do — which produces client-visible failures your own logs never mention. Add --trace=/tmp/doctor.jsonl for a raw dump of every request and response. See the Node CLI reference.

Node won't start: missing signing address

The node refuses to start because the signing address it expects doesn't match any mnemonic it loaded. The address it wants comes from zs.signing_addr (or, if you didn't set that, your on-chain operator record); the mnemonics come from the environment or your secret manager. Check, in order:

• The mnemonic actually reached the process. A variable set in your shell isn't set in the service's environment. For systemd, confirm the EnvironmentFile= points at a readable file that defines it.
• The mnemonic derives the right address. Decode it locally (any wallet that shows the derived address will do) and compare.
• The on-chain record names the address you think it does. Open the operator dashboard and check the node record's signing address.

See Encryption & keys for how the signing mnemonic is provisioned.

Reserve fails: payment not verified

Reserves are rejected with a payment-verification error, and the node's logs show it can't find the payer's transaction. This almost always means the algod your node talks to isn't observing pending transactions — so in the window between the payer broadcasting the escrow-open and the network confirming it, the node looks for that transaction and comes up empty.

The fix is to give the node an algod that sees the mempool. If you run your own non-participation node, set ForceFetchTransactions: true in its config.json and restart it. See Installation → your algod must observe the mempool for the details and the valid algod configurations.

warning

Tightening zs.mempool_poll_timeout / mempool_poll_interval only changes how the node spaces its lookups — it can't create visibility that isn't there. If your algod doesn't observe the mempool, no amount of polling will help.

The oracle is unavailable and prices show 0

/v1/zs/details reports oracle_healthy: false and algo_usd_price reads 0. This is non-fatal — your node still issues tickets and serves requests normally. The only effect is that, for that pricing cycle, the ALGO-network-fee figure clients display in USD is dropped, and the (optional) per-request minimum ALGO charge falls to zero. Token pricing and in-flight tickets are unaffected — those rates were pinned at reserve time.

Common causes:

• The ALGO/USD price feed is rate-limiting or down.
• Outbound network access to the feed is blocked.
• The node host's clock has drifted (the oracle compares timestamps).

If you don't charge the optional ALGO minimum and your clients don't need the fee-in-USD figure, an unavailable oracle is harmless. Otherwise, widen zs.oracle.max_staleness cautiously, or point zs.oracle.endpoint at a mirror you control (see Configuration).

Reserve returns a no-capacity 429

A model's admission pool is full: every slot is held by a reserved or in-flight request, so the node refuses new reserves with 429 no_capacity and a short Retry-After. Confirm it on metrics — zs_reserve_inflight_slots{model} has reached zs_reserve_capacity_slots{model} while zs_reserve_capacity_denied_total{model} climbs. Then either:

• Raise the model's max_active_tickets — and verify your backend and VRAM can actually handle the added concurrency (see Serving models).
• Lower zs.ticket_ttl so unused reservations recycle faster.
• Add another node serving the model so clients can spread the load.

This is separate from the rate limiter's 429 rate_limited, which throttles per-IP or per-account (see Configuration).

Tool loops end early, or reserves are refused

Answers arrive after one tool round when they should have taken several, or prompts come back refused with input_budget_exceeded. Both mean a request's measured input exceeded what its ticket reserved — see zs.reserve. Split them apart on the metric, which labels every occurrence by what happened:

curl -s localhost:9091/metrics | grep zs_reserve_input_budget_over_total

• outcome="tool_cutoff" — loops are hitting the ceiling. Your tool_headroom_per_iteration (default 4000) is undersized for the tools you serve: one web_read at the default 64 KiB max_bytes is roughly 16,000 tokens. Raise it, or lower zs.builtin_tools.web_read.max_bytes so results fit the budget you advertise. Lowering zs.builtin_tools.max_iterations without raising the headroom makes this worse, because the reserve clients compute is the product of the two.
• outcome="rejected" — plain requests are being refused before any upstream call, at zero cost to the payer. An honest client sizes with the same bound your node measures with, so isolated hits are one caller sizing badly. A broad, sustained rate across many payers is more likely a client older than your node's protocol version; confirm against the paired WARN line reserve input budget exceeded on initial body.
• outcome="monitored" — you have enforce_input_budget: false. Nothing is being refused or cut off; these requests were served at your expense. This is the mode to run temporarily while sizing, then turn enforcement back on.

If you need to serve while you tune, set enforce_input_budget: false to fall back to monitor mode — but treat it as temporary. It is what stops a payer under-declaring input_count and having you serve the difference, and the node logs a startup WARN for as long as it's off.

Clients say tools never fire, but the model supports them

Three different faults look identical from the outside, and zs-node doctor tells them apart — it reports tools accepted and tool calls emitted as separate lines, because they have different fixes.

• The backend rejects tools[] outright. On vLLM or SGLang that is a serving flag, not a model limitation: you need --enable-auto-tool-choice together with a --tool-call-parser matching the model's chat template. llama-server needs --jinja with a tool-declaring template. If your config also says context.tool_use: true, doctor reports it as a failure — the node is advertising a capability every client then gets a 4xx for.
• The backend accepts tools[] and never returns a call. Almost always a --tool-call-parser that doesn't match the template: the request is valid, the model emits a call in its own format, and the runtime doesn't recognise it.
• The backend accepts tools[] but rejects tool_choice: "required". Normal tool calling works; only clients that force a specific call get a 4xx. Doctor reports this as a warning rather than a failure, and says which form worked.

zs-node doctor --models=<your-model-id> --trace=/tmp/tools.jsonl
jq -c 'select(.step|startswith("tool_use")) | {step,status,response_body}' /tmp/tools.jsonl

Reasoning shows up in the backend but not through the node

Your backend visibly produces reasoning, but clients don't see it, or the model loses its thread across a tool call. Two independent settings are involved, and context.reasoning controls both.

context.reasoning.supported gates whether the node replays reasoning back to the model between tool-loop iterations. A reasoning model declared as supported: false still streams its reasoning to the client, but stops having it replayed — so it re-plans from scratch each iteration. This bites hardest on OpenAI's o-series and GPT-5 behind openai_passthrough, where nothing in the model list announces that the model reasons.

context.reasoning.allowed_efforts is what clients are offered, and the node forwards a client's chosen effort to your backend as reasoning_effort. Plenty of runtimes emit reasoning natively while rejecting that field as an unknown argument — so advertising an effort your backend doesn't take turns into a 400 for the client that picks it. If your backend is in that group, declare supported: true with no allowed_efforts.

zs-node doctor reports reasoning emitted and reasoning_effort accepted separately for exactly this reason, and flags either one disagreeing with your config.

Doctor says a model rejects image input, but it doesn't

zs-node doctor establishes image support by sending a small test image. If your backend refuses that particular image — a minimum-dimension rule, a format restriction, a decode failure — the exam quotes the upstream's own message alongside the finding. Read it: a complaint about the image ("invalid image", "dimensions below the minimum") is not the same as a complaint about the model ("this model does not support image input"), and only the second is a real verdict.

The probe tries two images and refuses to record a negative from wording it can't attribute to the capability, so this should be rare. If you see it anyway for a model you know takes images, --trace has the exact exchange:

zs-node doctor --models=<your-model-id> --trace=/tmp/vision.jsonl
jq -c 'select(.step|startswith("vision")) | {step,status,response_body}' /tmp/vision.jsonl

Please report the message — the classifier is a list of known wordings, and an unrecognized one is a gap worth closing. In the meantime the config wins: context.input_modalities is what the node advertises, and doctor's finding is advisory.

The local model keeps crashing

When you run a local model, its supervisor restarts a crashed engine up to five times in a minute, then gives up. Check:

• The model file path is correct and readable by the user the node runs as.
• gpu_layers doesn't exceed your VRAM — drop it, or use a smaller quantization.
• context_window × parallel_slots fits in VRAM — that product is your KV cache, and it is routinely larger than the weights. Do the arithmetic in What the inference backend needs.
• Your extra_args don't re-set values the node derives itself (context size, parallelism, continuous batching) — those are rejected at validation, and if one slips through it silently breaks the per-session contract.
• On dual-GPU without NVLink, layer-split mode is usually faster than row-split.

Settlement entries stuck in settling

A settle transaction was submitted but algod didn't confirm it in time, so the entry flips back to be retried automatically. Occasional churn here is normal. Entries that stay stuck point at one of:

• algod is unreachable — the node can't submit or confirm.
• The signing account is out of ALGO — no fee budget to settle. Top it up (see Operations).
• Network congestion — confirmations are slow but eventual.

A late confirmation isn't lost: the node reconciles in-flight settlements against the chain when it restarts, so a planned restart never drops ledger state. If you need to inspect or hand-repair a specific ticket, the settlement admin subcommand is the escape hatch.

Out of VRAM at startup

Your local engine won't allocate. Lower one of:

• the model's context_window,
• llm.local.parallel_slots (set it explicitly to override the auto-derived value),
• zs.max_active_tickets (which also lowers the auto-derived slot count),
• or gpu_layers (offloading some layers to CPU — slower, but it fits).

Failing that, use a smaller quantization or a smaller base model.

Clients get 502 or 504 errors, but the node looks fine

Anything you put in front of the node — CDN, ingress controller, reverse proxy, load balancer — can answer for it, and when it does the error comes from that layer, not from the node. Clients can tell the difference: the node's own refusals always carry a structured error code, a gateway's don't.

What a client does about it depends on when the error lands. While a request is still being set up, the client retries and reroutes automatically, so brief blips never reach the user — but they still cost you the traffic, and a client that keeps failing against you there demotes you in its routing for several minutes. Once a request is paid for and sent, there is no retry: a gateway error during inference goes straight back to the caller as a failed request.

Timeouts are the one class nobody gets demoted for. A client can't tell your gateway giving up from a relay's, so it declines to blame either — which means a timeout problem never shows up as a ranking penalty. It just keeps failing until you fix it.

Two causes, in order of likelihood:

Restart gaps. If your front door serves errors while the node is down, use graceful shutdown and point your readiness probe at the private listener — see Graceful shutdown. A draining node stops advertising before the gap opens, so clients route elsewhere and never meet the error.

Origin timeouts shorter than your model's thinking time. A reasoning-heavy request can send nothing for a long stretch before the first token. A gateway that gives up during that silence reports 504 (Cloudflare: 524) even though the node was working normally. How long a silence you have to survive depends on how the caller asked:

• Streamed requests — the node sends an invisible keepalive during silence (server.sse_keepalive_interval, 15s by default), which satisfies most intermediaries. Your origin read/idle timeout still has to exceed your slowest first-token time, since the keepalive only starts once the response has begun.
• Buffered (non-streaming) requests — there is no keepalive, and there can't be: nothing at all goes over the connection until the model has finished the whole reply. Your origin timeout has to exceed your slowest complete generation, which is a far bigger number. This is the usual cause of a 504 that lands almost exactly on a round minute.

You can't control which shape callers use, so size for the buffered case if you want to serve them reliably. Defaults that are commonly too low: nginx proxy_read_timeout (60s), AWS ALB idle timeout (60s), Cloudflare (~100s, and not raisable except on Enterprise — if your slowest generation exceeds it, that's a hard ceiling you have to design around rather than tune).

Errors in the 520–527 range are Cloudflare-specific and always mean the problem is between Cloudflare and your node — 521 origin down, 522 connection timed out, 523 origin unreachable, 524 origin timed out.

The node can't reach algod, the price feed, or the backend

Almost always an egress firewall. The node needs outbound HTTPS to:

• your algod endpoint,
• the ALGO/USD price feed (unless you've overridden zs.oracle.endpoint),
• and your inference backend's URL.

From inside the container or host, curl -v against each is the fastest way to find which one is blocked.

The node only listens on localhost

server.listen defaults to loopback (127.0.0.1:9090), so a node reachable only on localhost hasn't been exposed yet. Set NODE_SERVER_LISTEN=:9090 (dual-stack IPv4+IPv6; 0.0.0.0 would be IPv4-only) so the node faces clients directly, and serve HTTPS with tls.mode: acme (or terminate TLS at a single reverse proxy — never a load balancer; see Installation). The private listener (server.private_listen, default 127.0.0.1:9091, serving /healthz, /livez and /metrics): override it if your probes or scraper live in another container, or set it empty to colocate those routes on the public port. In YAML or compose, quote a bare ":9091" — the leading colon is otherwise read as a mapping.