1.9 KiB
ADR 0002 — Replica-Aware Routing with Atomic Reservations
Status: Accepted
Each block hash is associated with every backend believed to cache that prefix, rather than one overwritable owner. A follow-up chooses the least-loaded location of its longest known prefix; a first request or sub-block prompt chooses from all backends. Selection and load reservation occur atomically so concurrent requests cannot all select from the same stale load snapshot.
The Merkle chain identifies the complete rendered-prompt namespace, not only messages: it includes model, tool definitions, chat-template inputs such as chat_template_kwargs, and unknown prompt extensions while excluding known generation-only controls. A follow-up is identified by an assistant, tool, or legacy function message role.
Cache locations are committed only after a successful upstream response begins. Load has two phases: novel-block prefill cost is released at response headers, while one active-request unit remains until completion, error, or client disconnect. This preserves cache replicas, avoids poisoning locality state on failed requests, and accounts for generation work even when the prompt is fully cached.
Each block/backend association also has a configurable idle TTL. Routing prunes expired associations as it walks the prefix, and a successful request refreshes every association accessed on its selected backend. This bounds stale cache assumptions without requiring backend cooperation.
A background vacuum uses an expiry heap rather than scanning the full block map or polling on a fixed interval. It sleeps until the earliest deadline and is notified when an insertion moves that deadline earlier. Refreshing an association appends a new deadline; when an older deadline reaches the heap front, the vacuum compares it with the association's current deadline and ignores it if the location was refreshed. Empty block entries are removed with their final expired location.