sgl-project/sglang · error · RuntimeError

AttentionOffsetCache should not store data

Error message

AttentionOffsetCache should not store data

What it means

`AttentionOffsetCache` is a lightweight offset-tracking cache (used for mask computation) that intentionally holds no key/value data. Calling `update_and_fetch` on it is a programming error: the class only records sequence offsets, so there is nothing to fetch. The library raises RuntimeError to fail fast rather than silently return wrong results.

Source

Thrown at python/sglang/srt/hardware_backend/mlx/kv_cache/attention_kv_cache.py:59

    """Data-free shim satisfying mlx-lm's cache protocol.

    Provides ``make_mask`` and ``state`` without storing actual K/V.
    """

    def __init__(self, offset: int = 0):
        self.offset = offset

    @property
    def state(self):
        return ()  # Empty — safe for mx.eval unpacking

    def make_mask(self, N, return_array=False, window_size=None, **kwargs):
        return make_attention_mask(
            N, self.offset, return_array=return_array, window_size=window_size
        )

    def update_and_fetch(self, keys, values):
        raise RuntimeError("AttentionOffsetCache should not store data")


_DEFAULT_MAX_SEQ_LEN = 4096


class ContiguousAttentionKVCache:
    """Pre-allocated attention KV buffer for one request and one layer.

    Shape ``(1, n_kv_heads, max_seq_len, head_dim)``.  Slice assignment
    instead of ``mx.concatenate``.  Lazy-allocated on first write.
    """

    __slots__ = ("keys", "values", "offset", "max_seq_len")

    def __init__(
        self,
        n_kv_heads: int | None = None,
        head_dim: int | None = None,

View on GitHub (pinned to 0132848349)

Solutions

1. Use `ContiguousAttentionKVCache` (or another storage-backed class) where data storage is needed.
2. Add an isinstance/type check before calling update_and_fetch so offset-only caches take a different path.
3. Restructure the caller so mask generation (make_mask) is the only API used on AttentionOffsetCache.

Example fix

# before
cache.update_and_fetch(keys, values)  # RuntimeError if cache is AttentionOffsetCache

# after
if isinstance(cache, ContiguousAttentionKVCache):
    cache.update_and_fetch(keys, values)
else:
    cache.update(keys, values)  # offset-only path

from sglang.srt.hardware_backend.mlx.kv_cache.attention_kv_cache import AttentionOffsetCache
if isinstance(cache, AttentionOffsetCache):
    cache.update(keys, values)  # offset-only API
else:
    cache.update_and_fetch(keys, values)

def is_storage_backed(cache) -> bool:
    return hasattr(cache, "update_and_fetch") and not isinstance(cache, AttentionOffsetCache)

When it happens

Trigger: Passing an `AttentionOffsetCache` instance to code that calls `update_and_fetch(keys, values)` — e.g. `to_contiguous` or any generic KV-cache routine that assumes the full ContiguousAttentionKVCache interface.

Common situations: Refactoring code so an offset-only cache is used where a storage-backed cache is expected; writing generic utilities that accept 'any cache object' without checking the concrete type; mlx_lm-style code paths that call update_and_fetch unconditionally.

Related errors

• WindowedAttentionKVCache holds only the trailing window and
• BatchedDecodeContext requires full_kv_pool_index_by_layer wh
• Layer count and attention attribute count differ: {len(layer
• MLX model has no supported attention layers
• {self.__class__.__name__}.apply should not be called.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f6b62711914fc516. Report an issue: GitHub.