{"record":{"id":"f18c359403dee437","repo":"sgl-project/sglang","slug":"windowedattentionkvcache-holds-only-the-trailing-w","errorCode":null,"errorMessage":"WindowedAttentionKVCache holds only the trailing window and cannot serve a full-context attention mask","messagePattern":"WindowedAttentionKVCache holds only the trailing window and cannot serve a full-context attention mask","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/hardware_backend/mlx/kv_cache/attention_kv_cache.py","lineNumber":203,"sourceCode":"        self.offset = 0  # absolute: every token ever written\n        self._local = 0  # tokens currently in the buffer\n\n    @property\n    def state(self):\n        \"\"\"Arrays for ``mx.eval`` unpacking.\"\"\"\n        if self.keys is None:\n            return ()\n        return (self.keys, self.values)\n\n    def reset(self) -> None:\n        \"\"\"Reset for reuse, keeping allocated buffers.\"\"\"\n        self.offset = 0\n        self._local = 0\n\n    def make_mask(self, N, return_array=False, window_size=None, **kwargs):\n        kept = min(self._local, self.window)\n        if window_size is None and self.offset > kept:\n            raise RuntimeError(\n                \"WindowedAttentionKVCache holds only the trailing window and \"\n                \"cannot serve a full-context attention mask\"\n            )\n        # No N == 1 shortcut here: mlx_lm's banded mask is\n        # ``linds < rinds + window_size`` (strict), so a window of W admits\n        # exactly W keys, while this buffer returns W + 1 once ``kept ==\n        # window`` -- the trailing window plus the token just written.\n        return make_attention_mask(\n            N, kept, return_array=return_array, window_size=window_size\n        )\n\n    def _append(self, keys: mx.array, values: mx.array) -> tuple[int, int]:\n        \"\"\"Append a chunk in place; return the (start, end) span it serves.\n\n        Split out of ``update_and_fetch`` so the decode path can skip building\n        the two return slices, which it discards in favour of ``get_kv``.\n        \"\"\"\n        S = keys.shape[2]","sourceCodeStart":185,"sourceCodeEnd":221,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/hardware_backend/mlx/kv_cache/attention_kv_cache.py#L185-L221","documentation":"`WindowedAttentionKVCache` retains only the trailing `window` tokens, discarding older keys/values. A full-context (non-windowed) attention mask requires all historical positions, which the buffer no longer holds once `offset > kept`, so make_mask refuses rather than returning a silently-truncated mask. The check specifically fires when `window_size=None` (full attention) is requested.","triggerScenarios":"Calling `make_mask(N)` with no `window_size` on a WindowedAttentionKVCache whose sequence has grown past the window (`self.offset > min(self._local, self.window)`) — e.g. switching a sliding-window layer to full attention, or a long-prompt prefill overflowing the window.","commonSituations":"Using a sliding-window model (e.g. Gemma/Mistral-style) with a prompt longer than the window; code that assumes all layers can compute full attention; changing window sizes mid-generation or evaluating with full-context metrics.","solutions":["Pass the layer's actual window_size to make_mask (make_mask(N, window_size=layer_window)) instead of None.","Use a full (Contiguous) KV cache for layers/prompts that need full attention.","Cap input length to the window when full-context masks are required."],"exampleFix":"# before\nmask = cache.make_mask(N)  # window_size=None -> RuntimeError after window overflow\n\n# after\nmask = cache.make_mask(N, window_size=model_layer_window)","handlingStrategy":"validation","validationCode":"if window_size is None and cache.offset > min(getattr(cache, \"_local\", 0), getattr(cache, \"window\", 0)):\n    window_size = model_layer_window  # don't request a full-context mask\nmask = cache.make_mask(N, window_size=window_size)","typeGuard":"def can_serve_full_mask(cache) -> bool:\n    return cache.offset <= min(getattr(cache, \"_local\", 0), getattr(cache, \"window\", float(\"inf\")))","tryCatchPattern":null,"preventionTips":["Always pass the layer's window_size for sliding-window layers.","Use ContiguousAttentionKVCache when full-context attention is required.","Cap prompts to the window size for windowed models."],"tags":["mlx","kv-cache","sliding-window","attention-mask"],"backgroundTag":"context-window-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}