sgl-project/sglang · error · NotImplementedError

Use get_key_buffer instead.

Error message

Use get_key_buffer instead.

What it means

The DeepSeek-V4 single-KV pool stores keys and values in one shared kv_buffer and has no separate value buffer, so get_value_buffer and get_kv_buffer deliberately raise NotImplementedError telling callers to use get_key_buffer instead (set_kv_buffer is likewise not supported).

Source

Thrown at python/sglang/srt/mem_cache/deepseek_v4_memory_pool.py:171

        return fused_store_cache(
            input=cache_k,
            cache=self.kv_buffer[layer_id],
            indices=loc,
            page_size=self.page_size,
            type="flashmla",
        )

    def get_key_buffer(self, layer_id: int):
        if self.store_dtype != self.dtype:
            return self.kv_buffer[layer_id - self.start_layer].view(self.dtype)

        return self.kv_buffer[layer_id]

    def set_kv_buffer(self, *args, **kwargs) -> None:
        raise NotImplementedError()

    def get_value_buffer(self, layer_id: int) -> torch.Tensor:
        raise NotImplementedError("Use get_key_buffer instead.")

    def get_kv_buffer(self, layer_id: int) -> Tuple[torch.Tensor, torch.Tensor]:
        raise NotImplementedError("Use get_key_buffer instead.")


class HiSparseC4DevicePool(DeepSeekV4SingleKVPool):

    def __init__(
        self,
        size: int,
        page_size: int,
        dtype: torch.dtype,
        qk_nope_head_dim: int,
        qk_rope_head_dim: int,
        layer_num: int,
        device: str,
        enable_memory_saver: bool,
        start_layer: int | None = None,

View on GitHub (pinned to 0132848349)

Solutions

1. Replace get_value_buffer()/get_kv_buffer() calls with get_key_buffer(layer_id), which returns the shared kv_buffer for this pool
2. Add an isinstance/hasattr check before calling the generic accessors
3. Extend DeepSeekV4SingleKVPool in a subclass if you genuinely need a separate value view

Example fix

# before
k, v = pool.get_kv_buffer(layer_id)
# after
k = pool.get_key_buffer(layer_id)  # single-KV pool: shared buffer

if isinstance(pool, DeepSeekV4SingleKVPool):
    buf = pool.get_key_buffer(layer_id)
else:
    k, v = pool.get_kv_buffer(layer_id)

from sglang.srt.mem_cache.deepseek_v4_memory_pool import DeepSeekV4SingleKVPool

def is_single_kv_pool(pool) -> bool:
    return isinstance(pool, DeepSeekV4SingleKVPool) or type(pool).get_value_buffer is DeepSeekV4SingleKVPool.get_value_buffer

try:\n    k, v = pool.get_kv_buffer(layer_id)\nexcept NotImplementedError:\n    k = pool.get_key_buffer(layer_id)  # single-KV layout

When it happens

Trigger: Calling pool.get_value_buffer(layer_id) or pool.get_kv_buffer(layer_id) on a DeepSeekV4SingleKVPool / HiSparseC4DevicePool instance — e.g. generic attention backend or custom kernel code that assumes the standard MHATokenToKVPool interface.

Common situations: Porting an attention backend or profiler written against standard KV pools to DSv4's single-KV layout; third-party code calling get_kv_buffer unconditionally over all pool types.

Related errors

• {self.__class__.__name__}.apply should not be called.
• evictable_size() is not implemented; use full_evictable_size
• protected_size() is not implemented; use full_protected_size
• Missing previous frame for delta payload
• /v1/models ${response.status}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e2574b3655dd88f0. Report an issue: GitHub.