sgl-project/sglang · error · RuntimeError

WindowedAttentionKVCache holds only the trailing window and

Error message

WindowedAttentionKVCache holds only the trailing window and cannot serve a full-context attention mask

What it means

`WindowedAttentionKVCache` retains only the trailing `window` tokens, discarding older keys/values. A full-context (non-windowed) attention mask requires all historical positions, which the buffer no longer holds once `offset > kept`, so make_mask refuses rather than returning a silently-truncated mask. The check specifically fires when `window_size=None` (full attention) is requested.

Source

Thrown at python/sglang/srt/hardware_backend/mlx/kv_cache/attention_kv_cache.py:203

        self.offset = 0  # absolute: every token ever written
        self._local = 0  # tokens currently in the buffer

    @property
    def state(self):
        """Arrays for ``mx.eval`` unpacking."""
        if self.keys is None:
            return ()
        return (self.keys, self.values)

    def reset(self) -> None:
        """Reset for reuse, keeping allocated buffers."""
        self.offset = 0
        self._local = 0

    def make_mask(self, N, return_array=False, window_size=None, **kwargs):
        kept = min(self._local, self.window)
        if window_size is None and self.offset > kept:
            raise RuntimeError(
                "WindowedAttentionKVCache holds only the trailing window and "
                "cannot serve a full-context attention mask"
            )
        # No N == 1 shortcut here: mlx_lm's banded mask is
        # ``linds < rinds + window_size`` (strict), so a window of W admits
        # exactly W keys, while this buffer returns W + 1 once ``kept ==
        # window`` -- the trailing window plus the token just written.
        return make_attention_mask(
            N, kept, return_array=return_array, window_size=window_size
        )

    def _append(self, keys: mx.array, values: mx.array) -> tuple[int, int]:
        """Append a chunk in place; return the (start, end) span it serves.

        Split out of ``update_and_fetch`` so the decode path can skip building
        the two return slices, which it discards in favour of ``get_kv``.
        """
        S = keys.shape[2]

View on GitHub (pinned to 0132848349)

Solutions

1. Pass the layer's actual window_size to make_mask (make_mask(N, window_size=layer_window)) instead of None.
2. Use a full (Contiguous) KV cache for layers/prompts that need full attention.
3. Cap input length to the window when full-context masks are required.

Example fix

# before
mask = cache.make_mask(N)  # window_size=None -> RuntimeError after window overflow

# after
mask = cache.make_mask(N, window_size=model_layer_window)

if window_size is None and cache.offset > min(getattr(cache, "_local", 0), getattr(cache, "window", 0)):
    window_size = model_layer_window  # don't request a full-context mask
mask = cache.make_mask(N, window_size=window_size)

def can_serve_full_mask(cache) -> bool:
    return cache.offset <= min(getattr(cache, "_local", 0), getattr(cache, "window", float("inf")))

When it happens

Trigger: Calling `make_mask(N)` with no `window_size` on a WindowedAttentionKVCache whose sequence has grown past the window (`self.offset > min(self._local, self.window)`) — e.g. switching a sliding-window layer to full attention, or a long-prompt prefill overflowing the window.

Common situations: Using a sliding-window model (e.g. Gemma/Mistral-style) with a prompt longer than the window; code that assumes all layers can compute full attention; changing window sizes mid-generation or evaluating with full-context metrics.

Related errors

• kv-canary: SWA slot {slot} is outside full_to_swa_index_mapp
• recent_window_tokens must be non-negative or None
• recent_window_tokens must be >= 0 or None
• AttentionOffsetCache should not store data
• BatchedDecodeContext requires full_kv_pool_index_by_layer wh

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f18c359403dee437. Report an issue: GitHub.