Fri, 24 Jul 2026 16:15:47 +0800

CONTEXT.md

@@ -0,0 +1,78 @@
+# KVCA Proxy — Ubiquitous Language
+
+## Core Concepts
+
+### KV Cache-Aware Proxy
+An LLM gateway that routes requests to backend inference servers with awareness of KV cache state. The goal is to minimize recomputation by pinning conversational sequences to the same backend, while also choosing the best backend for initial requests based on a cache-locality utility metric.
+
+### First Request / Bootstrap Request
+A request whose `messages` array contains no prior model output or tool result. A request whose prompt identity is shorter than one block is treated the same way because it offers negligible reusable cache locality.
+
+### Follow-up Request
+A request whose `messages` array contains an `assistant`, `tool`, or legacy `function` role. The gateway routes it to a suitable cache location for its longest known prefix.
+
+### Backend
+An upstream LLM inference server (e.g., vLLM, TGI). The gateway is completely agnostic to the backend's internal caching — no custom protocol, headers, or cooperation is required.
+
+## Routing
+
+### Dispatch Phase 1 — Bootstrap Routing
+On a bootstrap request, the gateway computes the prompt CBR, builds its Merkle chain, and atomically reserves the least-loaded backend. Cache locations are learned only after that backend successfully begins responding.
+
+### Dispatch Phase 2 — Follow-up Routing
+On a follow-up request, the gateway walks the Merkle chain until its first block without an unexpired location. It atomically reserves the least-loaded replica of the preceding prefix and refreshes the selected location only after a successful upstream response begins.
+
+## Cache Tracking
+
+### Canonical Byte Representation (CBR)
+The deterministic byte encoding of every request input that can affect prompt rendering. It includes `messages`, model and tool configuration, chat-template options, and unknown extension fields, while excluding generation-only controls.
+
+### Block
+A fixed-size, complete segment of the canonical byte representation. Block size is configurable at startup (must be a multiple of 8). Partial blocks at the end of the CBR are discarded — they are unconditionally novel for any backend.
+
+### Merkle Block Chain
+Each complete block is content-addressed via SipHash:
+
+```
+hash[0] = SipHash(0_u64 || block[0])
+hash[i] = SipHash(hash[i-1] || block[i])
+```
+
+`block_size` must be a multiple of 8 so that each hash input (`prev` + `block[i]`) fills a whole number of 8-byte words with no trailing fragment. SipHash's `finish()` handles its own final padding and length suffix.
+
+### Global Block Map
+A best-effort association from each block hash to all unexpired backend cache locations for that prefix. A location is learned or refreshed only after the backend accepts and successfully begins the request.
+
+### Cache-Location TTL
+The maximum idle time for one backend's association with a cached block. An expired association is forgotten during routing; a successful access refreshes the associations for the selected prefix.
+
+### Cache-Location Vacuum
+Deadline-driven reclamation of expired cache-location associations, including removal of a block when it has no remaining locations. It complements expiry checks performed during routing.
+
+### Divergence Point
+The index of the first block in a follow-up request's Merkle chain with no known cache location. The locations associated with the preceding block are replicas of the longest known prefix.
+
+### Novel Block Cost
+The number of blocks from the divergence point to the end of the chain. These blocks will need to be computed from scratch by the selected backend.
+
+## Bookkeeping
+
+### Fallback Routing
+If the chain is empty or has no known prefix, every backend is eligible and the least-loaded backend is selected.
+
+### Staleness
+The gateway is optimistic between a location's last successful access and its TTL. A backend restart or early eviction can therefore cause a temporary suboptimal route, never an incorrect response.
+
+## Load Tracking
+
+### Novel-Block Load
+The estimated prefill work for blocks not already cached on the target backend. It is reserved during routing and released when the backend returns response headers.
+
+### Active-Request Load
+A baseline unit reserved for every request, including a fully cached follow-up. It remains reserved while the response is active and is released on completion, error, or client disconnect.
+
+### Least-Loaded Selection
+Routing selects the least-loaded eligible backend and reserves its estimated load as one indivisible decision. For a follow-up, eligible backends are the known replicas of the longest cached prefix.
+
+### Releasing Load
+The prefill portion is released when the backend returns headers. The active-request portion is released when the response completes, errors, or is abandoned by the client. A connection failure releases both portions.