{"record":{"id":"f6b62711914fc516","repo":"sgl-project/sglang","slug":"attentionoffsetcache-should-not-store-data","errorCode":null,"errorMessage":"AttentionOffsetCache should not store data","messagePattern":"AttentionOffsetCache should not store data","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/hardware_backend/mlx/kv_cache/attention_kv_cache.py","lineNumber":59,"sourceCode":"    \"\"\"Data-free shim satisfying mlx-lm's cache protocol.\n\n    Provides ``make_mask`` and ``state`` without storing actual K/V.\n    \"\"\"\n\n    def __init__(self, offset: int = 0):\n        self.offset = offset\n\n    @property\n    def state(self):\n        return ()  # Empty — safe for mx.eval unpacking\n\n    def make_mask(self, N, return_array=False, window_size=None, **kwargs):\n        return make_attention_mask(\n            N, self.offset, return_array=return_array, window_size=window_size\n        )\n\n    def update_and_fetch(self, keys, values):\n        raise RuntimeError(\"AttentionOffsetCache should not store data\")\n\n\n_DEFAULT_MAX_SEQ_LEN = 4096\n\n\nclass ContiguousAttentionKVCache:\n    \"\"\"Pre-allocated attention KV buffer for one request and one layer.\n\n    Shape ``(1, n_kv_heads, max_seq_len, head_dim)``.  Slice assignment\n    instead of ``mx.concatenate``.  Lazy-allocated on first write.\n    \"\"\"\n\n    __slots__ = (\"keys\", \"values\", \"offset\", \"max_seq_len\")\n\n    def __init__(\n        self,\n        n_kv_heads: int | None = None,\n        head_dim: int | None = None,","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/hardware_backend/mlx/kv_cache/attention_kv_cache.py#L41-L77","documentation":"`AttentionOffsetCache` is a lightweight offset-tracking cache (used for mask computation) that intentionally holds no key/value data. Calling `update_and_fetch` on it is a programming error: the class only records sequence offsets, so there is nothing to fetch. The library raises RuntimeError to fail fast rather than silently return wrong results.","triggerScenarios":"Passing an `AttentionOffsetCache` instance to code that calls `update_and_fetch(keys, values)` — e.g. `to_contiguous` or any generic KV-cache routine that assumes the full ContiguousAttentionKVCache interface.","commonSituations":"Refactoring code so an offset-only cache is used where a storage-backed cache is expected; writing generic utilities that accept 'any cache object' without checking the concrete type; mlx_lm-style code paths that call update_and_fetch unconditionally.","solutions":["Use `ContiguousAttentionKVCache` (or another storage-backed class) where data storage is needed.","Add an isinstance/type check before calling update_and_fetch so offset-only caches take a different path.","Restructure the caller so mask generation (make_mask) is the only API used on AttentionOffsetCache."],"exampleFix":"# before\ncache.update_and_fetch(keys, values)  # RuntimeError if cache is AttentionOffsetCache\n\n# after\nif isinstance(cache, ContiguousAttentionKVCache):\n    cache.update_and_fetch(keys, values)\nelse:\n    cache.update(keys, values)  # offset-only path","handlingStrategy":"type-guard","validationCode":"from sglang.srt.hardware_backend.mlx.kv_cache.attention_kv_cache import AttentionOffsetCache\nif isinstance(cache, AttentionOffsetCache):\n    cache.update(keys, values)  # offset-only API\nelse:\n    cache.update_and_fetch(keys, values)","typeGuard":"def is_storage_backed(cache) -> bool:\n    return hasattr(cache, \"update_and_fetch\") and not isinstance(cache, AttentionOffsetCache)","tryCatchPattern":null,"preventionTips":["Keep offset-only and storage caches in distinct type hierarchies.","Never pass AttentionOffsetCache into generic KV-store routines."],"tags":["mlx","kv-cache","api-misuse"],"backgroundTag":"unsupported-interface-method","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}