sgl-project/sglang · error · NotImplementedError

MHATokenToKOnlyPool does not allocate V

Error message

MHATokenToKOnlyPool does not allocate V

What it means

MHATokenToKOnlyPool is a K-only sparse pool that never allocates a V buffer; calling get_value_buffer on it is unsupported by design. V for these layers lives in the parent MiniMaxSparseKVPool's index-V pool, not here.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:4783

    def get_key_buffer(self, layer_id: int):
        if self.layer_transfer_counter is not None:
            self.layer_transfer_counter.wait_until(layer_id - self.start_layer)
        return self._get_key_buffer(layer_id)

    def set_k_buffer(
        self,
        layer_id: int,
        loc: torch.Tensor,
        cache_k: torch.Tensor,
    ) -> None:
        if cache_k.dtype != self.dtype:
            cache_k = cache_k.to(self.dtype)
        if self.store_dtype != self.dtype:
            cache_k = cache_k.view(self.store_dtype)
        self.k_buffer[layer_id][loc] = cache_k

    def get_value_buffer(self, layer_id: int) -> torch.Tensor:
        raise NotImplementedError("MHATokenToKOnlyPool does not allocate V")

    def get_kv_buffer(self, layer_id: int) -> Tuple[torch.Tensor, torch.Tensor]:
        raise NotImplementedError("MHATokenToKOnlyPool does not allocate V")

    def set_kv_buffer(
        self,
        layer: RadixAttention,
        loc: torch.Tensor,
        cache_k: torch.Tensor,
        cache_v: torch.Tensor,
        k_scale: Optional[float] = None,
        v_scale: Optional[float] = None,
        layer_id_override: Optional[int] = None,
    ) -> None:
        # Routed through MiniMaxSparseKVPool.set_index_k_buffer instead.
        raise NotImplementedError(
            "MHATokenToKOnlyPool: use set_index_k_buffer on the parent "
            "MiniMaxSparseKVPool — this pool does not store V"

View on GitHub (pinned to 0132848349)

Solutions

1. Use the parent MiniMaxSparseKVPool.get_index_kv_buffer / the pool API that routes V lookups to the index-V pool
2. Guard calls: only call get_value_buffer for layers with a V cache (check the layer id mapping)
3. Use get_key_buffer for K-only layers

Example fix

// before
v = pool.get_value_buffer(layer_id)  # pool is MHATokenToKOnlyPool
// after
k = pool.get_key_buffer(layer_id)
v = parent_sparse_pool.get_index_kv_buffer(index_kv_layer_id)

from sglang.srt.mem_cache.memory_pool import MHATokenToKOnlyPool
if isinstance(pool, MHATokenToKOnlyPool):
    raise TypeError('K-only pool has no V buffer; use parent sparse pool accessors')

def has_value_buffer(pool) -> bool:
    from sglang.srt.mem_cache.memory_pool import MHATokenToKOnlyPool
    return not isinstance(pool, MHATokenToKOnlyPool)

try:
    v = pool.get_value_buffer(layer_id)
except NotImplementedError:
    v = None  # K-only layer; V lives in the index-V pool

When it happens

Trigger: Calling pool.get_value_buffer(layer_id) where pool is (or delegates to) an MHATokenToKOnlyPool — e.g. generic code that assumes every KV pool can return both K and V buffers.

Common situations: Running MiniMax-style sparse-attention models with generic memory-pool tooling, metrics collection, or custom attention backends that iterate get_value_buffer over all layers.

Related errors

• MHATokenToKOnlyPool: use set_index_k_buffer on the parent Mi
• layer_id={layer_id} does not have an index V cache (either d
• layer_id={layer_id} is not a sparse attention layer; sparse
• layer.layer_id={layer.layer_id} does not have an index V cac
• layer.layer_id={layer.layer_id} is not in the K-only sparse

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5bdcab7958c6ce30. Report an issue: GitHub.