{"record":{"id":"64789eaa9358e13e","repo":"sgl-project/sglang","slug":"noopmhatokentokvpool-set-kv-buffer-was-called-thi","errorCode":null,"errorMessage":"NoOpMHATokenToKVPool.set_kv_buffer was called. This pool is only valid in prefill-only modes (e.g. --is-embedding, scoring) with the FA backend's fa_skip_kv_cache path active; the attention backend must never write to it. Check that the workload truly performs no decode and that the FA backend's fa_skip_kv_cache preconditions are met.","messagePattern":"NoOpMHATokenToKVPool\\.set_kv_buffer was called\\. This pool is only valid in prefill-only modes \\(e\\.g\\. --is-embedding, scoring\\) with the FA backend's fa_skip_kv_cache path active; the attention backend must never write to it\\. Check that the workload truly performs no decode and that the FA backend's fa_skip_kv_cache preconditions are met\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/memory_pool.py","lineNumber":3004,"sourceCode":"        placeholder_bytes = (\n            2\n            * self.layer_num\n            * self.page_size\n            * self.head_num\n            * max(self.head_dim, self.v_head_dim)\n            * self.store_dtype.itemsize\n        )\n        logger.info(\n            f\"KV Cache skipped (no-op pool). Logical #tokens: {num_tokens}, \"\n            f\"physical K/V size: ~{placeholder_bytes / 1024:.1f} KB placeholder\"\n        )\n\n    def get_kv_size_bytes(self):\n        # Report zero so downstream memory accounting matches reality.\n        return (0, 0)\n\n    def set_kv_buffer(self, *args, **kwargs):\n        raise RuntimeError(\n            \"NoOpMHATokenToKVPool.set_kv_buffer was called. This pool is only \"\n            \"valid in prefill-only modes (e.g. --is-embedding, scoring) with \"\n            \"the FA backend's fa_skip_kv_cache path active; the attention \"\n            \"backend must never write to it. Check that the workload truly \"\n            \"performs no decode and that the FA backend's fa_skip_kv_cache \"\n            \"preconditions are met.\"\n        )\n\n    def get_key_buffer(self, layer_id: int):\n        # Return the placeholder. The FA backend reads this before taking the\n        # fa_skip_kv_cache branch (which does not use it); the placeholder shape\n        # is (page_size, head_num, head_dim) so downstream .view() calls succeed.\n        return self.k_buffer[layer_id - self.start_layer]\n\n    def get_value_buffer(self, layer_id: int):\n        return self.v_buffer[layer_id - self.start_layer]\n\n    def get_kv_buffer(self, layer_id: int):","sourceCodeStart":2986,"sourceCodeEnd":3022,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/memory_pool.py#L2986-L3022","documentation":"NoOpMHATokenToKVPool is a zero-size placeholder KV pool used in prefill-only modes (embedding servers, scoring) where the FlashAttention backend runs with fa_skip_kv_cache so no KV data is ever written. If any code path calls set_kv_buffer on it, it means the attention backend is attempting a real KV write, which the pool cannot store, so it raises immediately rather than silently corrupting behavior.","triggerScenarios":"Running with --is-embedding or a scoring workload with the NoOp pool allocated, but the attention backend selected does not use the FA fa_skip_kv_cache path (e.g. a different backend, or decode/batch steps get scheduled), causing a normal set_kv_buffer call to hit the no-op pool.","commonSituations":"Mixing --is-embedding with an attention backend other than FlashAttention; sending a chat/generate request that triggers decode to an embedding server; FA version where fa_skip_kv_cache preconditions (e.g. no radix cache, prefill-only scheduler) are not met.","solutions":["Verify the workload is truly prefill-only: no decode steps, no generate/chat completions on an --is-embedding server","Ensure the attention backend is FlashAttention with fa_skip_kv_cache active (check server_args resolution and FA version support)","Check that fa_skip_kv_cache preconditions hold (e.g. radix cache disabled / no prefix caching in this mode)","If decode is actually required, do not use the NoOp pool: run without --is-embedding / scoring mode so a real KV pool is allocated"],"exampleFix":"# before\nserver_args = ServerArgs(is_embedding=True, attention_backend=\"triton\")\n# after\nserver_args = ServerArgs(is_embedding=True, attention_backend=\"fa\")  # enables fa_skip_kv_cache path","handlingStrategy":"validation","validationCode":"from sglang.srt.server_args import ServerArgs\nargs = ServerArgs.from_cli_args()\nassert args.attention_backend in (None, 'fa'), 'NoOp pool requires FA backend with fa_skip_kv_cache'\nassert not args.enable_radix_cache or args.is_embedding, 'check prefill-only preconditions'","typeGuard":"def is_noop_kv_pool(pool) -> bool:\n    return type(pool).__name__ == 'NoOpMHATokenToKVPool'","tryCatchPattern":"try:\n    pool.set_kv_buffer(layer, loc, k, v)\nexcept RuntimeError as e:\n    if 'NoOpMHATokenToKVPool' in str(e):\n        raise ConfigError('Decode attempted on prefill-only/embedding server') from e\n    raise","preventionTips":["Smoke-test an embedding server with a tiny generate request before production","Assert the resolved attention backend is FA before allocating the NoOp pool","Log the pool class chosen at startup so misconfigurations are visible"],"tags":["kv-cache","attention-backend","embedding","prefill-only"],"backgroundTag":"unsupported-operation-guard","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}