Skip to main content

Operations

Once your node is live and serving, the routine work is small: keep the signing account funded, change pricing or models when you want to, rotate keys on your own schedule, and upgrade cleanly. This page covers each of those.

Most config changes take effect on restart. In-flight tickets always carry the rate and terms they were quoted at, so a restart never changes the price of a request already under way — only new reserves see new settings. Run the node under a supervisor (systemd, Docker --restart, an orchestrator) so "restart" is a one-liner; see Installation.

Top up the signing account​

Your hot signing account fee-pays every on-chain action the node takes — every settlement, plus ACME and IP-sync updates if you've enabled them. It needs an active ALGO float, not just the minimum balance for your on-chain boxes.

  • Budget roughly (daily request volume × ~7,000 µALGO) + a few days of buffer.
  • Watch it automatically. The node polls the balance and publishes zs_signing_balance_algos, warning below your threshold (default 5 ALGO) and refusing to start below the 1 ALGO floor. Alert on the metric or the signing-address ALGO balance is low log line — see Monitoring.
  • Refill from the owner address. USDC settlement income lands at your owner address, not the signing address, so topping up means an off-chain hop (swap USDC→ALGO and transfer, or a manual send) on whatever cadence your volume needs.

The economics of this are covered in Staking & economics.

Update pricing​

Edit the rates​

Change zs.default_pricing and/or the pricing block on any zs.models[<id>] entry in your config. Rates are quoted in USD per million tokens (see Serving models & pricing).

Restart​

In-flight tickets keep their pinned rate; only new reserves see the new price.

Confirm​

Check your live catalog on /v1/zs/details (or in the model picker) to verify the new rate is being advertised.

Add or change a model​

Declare it​

Add a zs.models[<id>] entry. With zs.default_pricing set you don't need a pricing block unless you want a per-model rate; otherwise add explicit rates (input_rate: 0 / output_rate: 0 makes it deliberately free). Add a context block too — it's recommended, and required for a local model.

Wire the backend​

For a local model, add the corresponding llm.local.models entry (a node hosts one local model by design — see Serving models & pricing for multi-model hosts). For an OpenAI-compatible or Vertex backend, availability is governed by the backend's own model list; here you're just declaring pricing and capacity.

Restart​

The change is picked up on restart, and clients see the new model on their next discovery probe — typically within a minute (see Health & compatibility).

To stop serving a model, remove its entry and restart — drop it from the catalog so clients don't route to something you no longer run.

Update context windows and concurrency​

Both are per-model edits that take effect on restart:

  • Context window — edit context_window in the model's context block. The proxy reads the new capacity from your details document on its next refresh.
  • Output ceiling — edit max_output_tokens in the same block, keeping it below context_window. If you raise the window, revisit this too: it does not scale with it, and a stale low ceiling silently truncates long answers. See Set an output ceiling.
  • Concurrency cap — set max_active_tickets on the model to give it its own admission pool, or omit it to inherit the node default. Each model's in-flight count is tracked independently, so raising one model's cap doesn't touch the others. Raise it alongside real backend/VRAM headroom, or you'll just move a 429 no_capacity into an upstream error.

Rotate the signing key​

The signing key is a hot key — rotate it whenever you'd rotate any production credential.

Create the new key​

Generate a new mnemonic and address in your wallet.

Add it to the keystore​

Add the new mnemonic to your keystore source (env var or secret manager) — keep the old one for now so in-flight tickets can still be settled.

Update the node record​

From the operator dashboard, connect your owner wallet and update the node record's signing address to the new one. (Only the operator owner authorizes this.) See Registering on-chain.

Restart and verify​

The node picks up the new on-chain signing address, matches it to the new mnemonic, and starts signing with it. Once old in-flight tickets have settled, remove the old mnemonic from the keystore.

Rotate the encryption key — nothing to do​

There is no operator-managed encryption key to rotate. The node generates a short-lived encryption recipient in memory, signs it with your signing key, advertises it, and regenerates it automatically on a fixed cadence — keeping the previous one decryptable for a brief overlap so in-flight sealed requests still open. There's no file, no on-chain transaction, and nothing persisted (which is what gives forward secrecy). See Encryption & keys for how it works and what a client does when it races a rotation.

Rotate the owner account​

Less common, but supported. The owner address is where all your USDC settlement income lands, so treat this like moving a treasury account. From the operator dashboard, connect the current owner wallet and update the operator record's owner address — only the current owner can authorize it, and every future rotation must then come from the new owner account. The node signs with the hot signing key, not the owner key, so a running node needs no restart for an owner change (unless you've pinned zs.owner_addr in config, in which case update it to match and restart).

Upgrade the node​

Pull the new version​

Fetch the new binary or container image.

Restart​

Send SIGTERM (what systemd, Docker, and Kubernetes all do) and the node drains gracefully — see Graceful shutdown below. The settlement driver reconciles any in-flight settlements against the chain at startup, so a planned restart never loses ledger state.

Keep your node software current — a node on an incompatible protocol version is still up and serving, but clients drop it from selection (see Health & compatibility). After upgrading, confirm your monitoring targets still point at the private listener for /healthz, /livez, and /metrics (see Monitoring).

Graceful shutdown​

On SIGTERM the node stops accepting new work but always finishes the inference it already accepted. It does that in three stages:

  1. Immediately it stops advertising models, refuses new reservations, sheds new relay circuits, and reports unhealthy on /healthz. Clients re-checking during your restart route to a different operator instead of paying to reserve against a node that is about to exit.
  2. For server.drain_grace (default 20s) it still honors reservations issued before the shutdown began, so a request already on its way to you lands instead of losing the payment it has already committed.
  3. For up to server.drain_timeout (default 5m) it waits for the inference already running to finish. Nobody's paid-for response gets cut off mid-answer.

Then server.shutdown_timeout (default 30s) closes the remaining connections.

If you sent the SIGTERM, a second one skips the wait when you need the process gone immediately. That shortcut applies only to a shutdown you signalled: when the node starts draining on its own — because your operator was evicted or unregistered on-chain — it deliberately ignores your first SIGTERM so a routine stop can't truncate inference that's still running, and it runs to drain_timeout. Use SIGKILL if you have to cut that short.

warning

Four things must line up or the drain is defeated. Every layer that can kill a connection has to be sized for it — getting the supervisor right and the load balancer wrong buys you nothing.

  • Your supervisor's kill deadline must be longer than the total drain — drain_grace + drain_timeout + shutdown_timeout, 5m50s with the defaults. If you installed the service with zs-node install-service, this is already set for you (systemd TimeoutStopSec=400, launchd ExitTimeOut=400). You only need to set it by hand for a hand-written unit, a container, or Kubernetes: terminationGracePeriodSeconds: 400, docker run --stop-timeout 400, compose stop_grace_period: 400s. Watch the Docker defaults especially — both --stop-timeout and stop_grace_period default to 10 seconds, and Kubernetes to 30. And you must raise it everywhere if you raise drain_timeout. Otherwise your supervisor force-kills the node in the middle of draining and you lose the benefit.
  • Whatever fronts the node needs a drain window too. A reverse proxy or managed load balancer has its own connection-draining / deregistration delay, and it will cut a response mid-stream while the node is still dutifully draining. Size it to outlive drain_grace + drain_timeout + shutdown_timeout, and never stop or reload the front door before the node has finished draining. Some platforms disable connection draining by default, which drops everything in flight the instant the node leaves the pool — check yours rather than assuming. The order you want is: node drain < front-door drain < platform kill deadline.
  • Your readiness probe must target the private listener, where /healthz lives (see Monitoring). That's what pulls you out of the load balancer in stage 1. Without it, your own front door keeps forwarding requests to a node that is shutting down and answers them with a 502 or 504.
  • Point your liveness probe at /livez, not /healthz. /healthz reports unhealthy for the whole drain — that's the point of it — so a liveness check on it tells your orchestrator to restart the node in the middle of draining, killing the paid work the drain exists to finish. On an ordinary restart this stays hidden, but not when the node starts draining on its own (because your operator was evicted or unregistered on-chain): nothing is terminating the container, so the liveness probe just keeps failing and restarts it. If your node build has no /livez yet, drop the liveness probe instead.
info

On managed Kubernetes, the platform caps your grace period. It enforces its own ceiling during node upgrades and scale-downs regardless of what your Pod spec asks for, so the whole drain budget has to fit underneath it — which is the real limit on how large drain_timeout can usefully be. Preemptible / Spot instances get a drastically shorter ceiling (tens of seconds), too short to honor even the default drain, which makes them a poor fit for a serving node. Check your platform's published limits before raising drain_timeout, and if it offers an annotation to exempt long-running Pods from autoscaler eviction, use it.

Why it matters beyond tidiness: a node that dies seconds after issuing a reservation leaves that payer's USDC locked in escrow until the inactivity refund window elapses — for work you never performed. Rolling a fleet without draining does this to every payer who happened to reserve in the last seconds of each node's life.

Back up what matters​

Two things are worth backing up:

  • The signing mnemonic — back it up like any wallet seed (secret manager, paper, hardware safe). There's no encryption-key file to back up; that key is ephemeral and disposable by design.

  • The settlement ledger — the local SQLite database at zs.settlement_db_path. The contract's inactivity backstop protects payer funds even without it, but the ledger is your only record of in-flight obligations. A daily copy is plenty:

    sqlite3 /var/lib/zs-node/settlement.db \
      ".backup /backups/settlement-$(date +%F).db"