{"record":{"id":"995dcf930e370df5","repo":"sgl-project/sglang","slug":"the-hpc-ops-attention-backend-with-an-fp8-e4m3-kv","errorCode":null,"errorMessage":"The hpc_ops attention backend with an fp8_e4m3 KV cache requires the model to run the fused HPC-Ops QKNorm+RoPE+quant+StoreKV op (fused_qk_rope_store_kv_fp8), which produces the per-token-per-head Q scales. This is currently wired for HunYuan V3 only; use --kv-cache-dtype bfloat16 for other models.","messagePattern":"The hpc_ops attention backend with an fp8_e4m3 KV cache requires the model to run the fused HPC-Ops QKNorm\\+RoPE\\+quant\\+StoreKV op \\(fused_qk_rope_store_kv_fp8\\), which produces the per-token-per-head Q scales\\. This is currently wired for HunYuan V3 only; use --kv-cache-dtype bfloat16 for other models\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/hpc_ops_backend.py","lineNumber":528,"sourceCode":"        # is 1), unlike attention_decode's ``mtp`` which counts extra draft\n        # tokens (plain decode is 0). Passing 0 here makes the scheduler\n        # kernel launch with an invalid configuration.\n        hpc.assign_attention_decode_task(\n            cache_seqlens,\n            self._decode_task_map,\n            self.num_kv_heads,\n            mtp=1,\n            new_kv_included=True,\n            min_process_len=_DYNAMIC_SCHED_MIN_PROCESS_LEN,\n        )\n        return self._decode_task_map\n\n    def _take_fp8_scales(self, metadata: HPCOpsMetadata):\n        \"\"\"Pop the per-layer FP8 scales written by the fused RoPE op.\"\"\"\n        q_scale = metadata.hpc_q_scale\n        split_k_flag = metadata.hpc_split_k_flag\n        if q_scale is None:\n            raise RuntimeError(\n                \"The hpc_ops attention backend with an fp8_e4m3 KV cache \"\n                \"requires the model to run the fused HPC-Ops \"\n                \"QKNorm+RoPE+quant+StoreKV op (fused_qk_rope_store_kv_fp8), \"\n                \"which produces the per-token-per-head Q scales. This is \"\n                \"currently wired for HunYuan V3 only; use \"\n                \"--kv-cache-dtype bfloat16 for other models.\"\n            )\n        metadata.hpc_q_scale = None\n        metadata.hpc_split_k_flag = None\n        return q_scale, split_k_flag\n\n    def forward_extend(\n        self,\n        q: torch.Tensor,\n        k: torch.Tensor,\n        v: torch.Tensor,\n        layer: RadixAttention,\n        forward_batch: ForwardBatch,","sourceCodeStart":510,"sourceCodeEnd":546,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/hpc_ops_backend.py#L510-L546","documentation":"With an fp8_e4m3 KV cache, the hpc_ops backend needs per-token-per-head Q scales that are only produced by the fused HPC-Ops QKNorm+RoPE+quant+StoreKV op (fused_qk_rope_store_kv_fp8). That fused op is currently wired only into the HunYuan V3 model path, so if the forward metadata lacks hpc_q_scale (it is None), this RuntimeError is raised in _take_fp8_scales during forward_extend/forward_decode.","triggerScenarios":"Running --attention-backend hpc_ops together with --kv-cache-dtype fp8_e4m3 on any model other than HunYuan V3 (or a model that does not call the fused_qk_rope_store_kv_fp8 op), so metadata.hpc_q_scale is None when attention runs.","commonSituations":"User enables fp8 KV cache to save memory and picks hpc_ops for speed on a non-HunYuan model; the combination is unsupported because no other model emits the required Q scales.","solutions":["Run with --kv-cache-dtype bfloat16 (or fp8 disabled) when using hpc_ops on non-HunYuan models","Keep fp8_e4m3 KV cache but switch to an attention backend that supports fp8 KV without the fused op (e.g. flashinfer)","If you are porting a model, wire fused_qk_rope_store_kv_fp8 into its QKNorm+RoPE path so hpc_q_scale is populated"],"exampleFix":"# before\npython -m sglang.launch_server --model Qwen2.5-72B --attention-backend hpc_ops --kv-cache-dtype fp8_e4m3\n# after\npython -m sglang.launch_server --model Qwen2.5-72B --attention-backend hpc_ops --kv-cache-dtype bfloat16","handlingStrategy":"validation","validationCode":"def fp8_ok_with_hpc_ops(model_arch: str) -> bool:\n    return model_arch in ('HunYuanV3ForCausalLM',)  # archs wired to fused_qk_rope_store_kv_fp8\n\nif server_args.kv_cache_dtype == 'fp8_e4m3' and not fp8_ok_with_hpc_ops(model_arch):\n    server_args.kv_cache_dtype = 'bfloat16'","typeGuard":null,"tryCatchPattern":"try:\n    out = backend.forward_extend(...)\nexcept RuntimeError as e:\n    if 'fused_qk_rope_store_kv_fp8' in str(e):\n        restart_with_kv_dtype('bfloat16')\n    else:\n        raise","preventionTips":["Treat hpc_ops+fp8 KV as HunYuan-only","Validate server_args combos in a startup sanity-check script","Watch release notes for newly supported fp8 fused-op models"],"tags":["hpc-ops","fp8","kv-cache-dtype","hunyuan","attention-backend","sglang"],"backgroundTag":"incompatible-backend-dtype-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}