sgl-project/sglang · critical · RuntimeError

NoOpMHATokenToKVPool.set_kv_buffer was called. This pool is

Error message

NoOpMHATokenToKVPool.set_kv_buffer was called. This pool is only valid in prefill-only modes (e.g. --is-embedding, scoring) with the FA backend's fa_skip_kv_cache path active; the attention backend must never write to it. Check that the workload truly performs no decode and that the FA backend's fa_skip_kv_cache preconditions are met.

What it means

NoOpMHATokenToKVPool is a zero-size placeholder KV pool used in prefill-only modes (embedding servers, scoring) where the FlashAttention backend runs with fa_skip_kv_cache so no KV data is ever written. If any code path calls set_kv_buffer on it, it means the attention backend is attempting a real KV write, which the pool cannot store, so it raises immediately rather than silently corrupting behavior.

Source

Thrown at python/sglang/srt/mem_cache/memory_pool.py:3004

        placeholder_bytes = (
            2
            * self.layer_num
            * self.page_size
            * self.head_num
            * max(self.head_dim, self.v_head_dim)
            * self.store_dtype.itemsize
        )
        logger.info(
            f"KV Cache skipped (no-op pool). Logical #tokens: {num_tokens}, "
            f"physical K/V size: ~{placeholder_bytes / 1024:.1f} KB placeholder"
        )

    def get_kv_size_bytes(self):
        # Report zero so downstream memory accounting matches reality.
        return (0, 0)

    def set_kv_buffer(self, *args, **kwargs):
        raise RuntimeError(
            "NoOpMHATokenToKVPool.set_kv_buffer was called. This pool is only "
            "valid in prefill-only modes (e.g. --is-embedding, scoring) with "
            "the FA backend's fa_skip_kv_cache path active; the attention "
            "backend must never write to it. Check that the workload truly "
            "performs no decode and that the FA backend's fa_skip_kv_cache "
            "preconditions are met."
        )

    def get_key_buffer(self, layer_id: int):
        # Return the placeholder. The FA backend reads this before taking the
        # fa_skip_kv_cache branch (which does not use it); the placeholder shape
        # is (page_size, head_num, head_dim) so downstream .view() calls succeed.
        return self.k_buffer[layer_id - self.start_layer]

    def get_value_buffer(self, layer_id: int):
        return self.v_buffer[layer_id - self.start_layer]

    def get_kv_buffer(self, layer_id: int):

View on GitHub (pinned to 0132848349)

Solutions

Verify the workload is truly prefill-only: no decode steps, no generate/chat completions on an --is-embedding server
Ensure the attention backend is FlashAttention with fa_skip_kv_cache active (check server_args resolution and FA version support)
Check that fa_skip_kv_cache preconditions hold (e.g. radix cache disabled / no prefix caching in this mode)
If decode is actually required, do not use the NoOp pool: run without --is-embedding / scoring mode so a real KV pool is allocated

Example fix

# before
server_args = ServerArgs(is_embedding=True, attention_backend="triton")
# after
server_args = ServerArgs(is_embedding=True, attention_backend="fa")  # enables fa_skip_kv_cache path

Defensive patterns

Strategy: validation

Validate before calling

from sglang.srt.server_args import ServerArgs
args = ServerArgs.from_cli_args()
assert args.attention_backend in (None, 'fa'), 'NoOp pool requires FA backend with fa_skip_kv_cache'
assert not args.enable_radix_cache or args.is_embedding, 'check prefill-only preconditions'

Type guard

def is_noop_kv_pool(pool) -> bool:
    return type(pool).__name__ == 'NoOpMHATokenToKVPool'

Try / catch

try:
    pool.set_kv_buffer(layer, loc, k, v)
except RuntimeError as e:
    if 'NoOpMHATokenToKVPool' in str(e):
        raise ConfigError('Decode attempted on prefill-only/embedding server') from e
    raise

Prevention

Smoke-test an embedding server with a tiny generate request before production
Assert the resolved attention backend is FA before allocating the NoOp pool
Log the pool class chosen at startup so misconfigurations are visible

When it happens

Trigger: Running with --is-embedding or a scoring workload with the NoOp pool allocated, but the attention backend selected does not use the FA fa_skip_kv_cache path (e.g. a different backend, or decode/batch steps get scheduled), causing a normal set_kv_buffer call to hit the no-op pool.

Common situations: Mixing --is-embedding with an attention backend other than FlashAttention; sending a chat/generate request that triggers decode to an embedding server; FA version where fa_skip_kv_cache preconditions (e.g. no radix cache, prefill-only scheduler) are not met.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/64789eaa9358e13e. Report an issue: GitHub.