Fri, 24 Jul 2026 16:13:44 +0800

docs/adr/0001-static-backend-config.md

@@ -0,0 +1,38 @@
+# ADR 0001 — Static Backend Configuration
+
+**Status**: Draft  
+**Date**: 2026-07-23
+
+## Context
+
+The gateway needs to know which upstream inference servers to route requests to. During initial development, the set of backends is known at deploy time.
+
+## Decision
+
+Backends are configured via a static TOML config file. The binary reads `config.toml` from the working directory by default, or accepts another path through `--config <PATH>`.
+
+```toml
+[[backend]]
+name = "gpu-01"
+address = "http://10.0.1.10:8000"
+
+[[backend]]
+name = "gpu-02"
+address = "http://10.0.1.11:8000"
+
+[cache]
+block_size = 512
+expiry_seconds = 300
+```
+
+`name` is a human-readable identifier for logging/metrics. `address` is the backend base URL. Cache block size and cache-location idle expiry are global startup settings.
+
+## Consequences
+
+- Simple, no external dependencies.
+- Adding/removing backends requires a config change and restart.
+- Dynamic registration can be added later.
+
+## Alternatives Considered
+
+- CLI flags per backend, environment variables, dynamic API.

docs/adr/0002-replica-aware-routing-reservations.md

@@ -0,0 +1,13 @@
+# ADR 0002 — Replica-Aware Routing with Atomic Reservations
+
+**Status**: Accepted
+
+Each block hash is associated with every backend believed to cache that prefix, rather than one overwritable owner. A follow-up chooses the least-loaded location of its longest known prefix; a first request or sub-block prompt chooses from all backends. Selection and load reservation occur atomically so concurrent requests cannot all select from the same stale load snapshot.
+
+The Merkle chain identifies the complete rendered-prompt namespace, not only `messages`: it includes model, tool definitions, chat-template inputs such as `chat_template_kwargs`, and unknown prompt extensions while excluding known generation-only controls. A follow-up is identified by an `assistant`, `tool`, or legacy `function` message role.
+
+Cache locations are committed only after a successful upstream response begins. Load has two phases: novel-block prefill cost is released at response headers, while one active-request unit remains until completion, error, or client disconnect. This preserves cache replicas, avoids poisoning locality state on failed requests, and accounts for generation work even when the prompt is fully cached.
+
+Each block/backend association also has a configurable idle TTL. Routing prunes expired associations as it walks the prefix, and a successful request refreshes every association accessed on its selected backend. This bounds stale cache assumptions without requiring backend cooperation.
+
+A background vacuum uses an expiry heap rather than scanning the full block map or polling on a fixed interval. It sleeps until the earliest deadline and is notified when an insertion moves that deadline earlier. Refreshing an association appends a new deadline; when an older deadline reaches the heap front, the vacuum compares it with the association's current deadline and ignores it if the location was refreshed. Empty block entries are removed with their final expired location.