sgl-project/sglang · error · RuntimeError

The hpc_ops attention backend with an fp8_e4m3 KV cache requ

Error message

The hpc_ops attention backend with an fp8_e4m3 KV cache requires the model to run the fused HPC-Ops QKNorm+RoPE+quant+StoreKV op (fused_qk_rope_store_kv_fp8), which produces the per-token-per-head Q scales. This is currently wired for HunYuan V3 only; use --kv-cache-dtype bfloat16 for other models.

What it means

With an fp8_e4m3 KV cache, the hpc_ops backend needs per-token-per-head Q scales that are only produced by the fused HPC-Ops QKNorm+RoPE+quant+StoreKV op (fused_qk_rope_store_kv_fp8). That fused op is currently wired only into the HunYuan V3 model path, so if the forward metadata lacks hpc_q_scale (it is None), this RuntimeError is raised in _take_fp8_scales during forward_extend/forward_decode.

Source

Thrown at python/sglang/srt/layers/attention/hpc_ops_backend.py:528

        # is 1), unlike attention_decode's ``mtp`` which counts extra draft
        # tokens (plain decode is 0). Passing 0 here makes the scheduler
        # kernel launch with an invalid configuration.
        hpc.assign_attention_decode_task(
            cache_seqlens,
            self._decode_task_map,
            self.num_kv_heads,
            mtp=1,
            new_kv_included=True,
            min_process_len=_DYNAMIC_SCHED_MIN_PROCESS_LEN,
        )
        return self._decode_task_map

    def _take_fp8_scales(self, metadata: HPCOpsMetadata):
        """Pop the per-layer FP8 scales written by the fused RoPE op."""
        q_scale = metadata.hpc_q_scale
        split_k_flag = metadata.hpc_split_k_flag
        if q_scale is None:
            raise RuntimeError(
                "The hpc_ops attention backend with an fp8_e4m3 KV cache "
                "requires the model to run the fused HPC-Ops "
                "QKNorm+RoPE+quant+StoreKV op (fused_qk_rope_store_kv_fp8), "
                "which produces the per-token-per-head Q scales. This is "
                "currently wired for HunYuan V3 only; use "
                "--kv-cache-dtype bfloat16 for other models."
            )
        metadata.hpc_q_scale = None
        metadata.hpc_split_k_flag = None
        return q_scale, split_k_flag

    def forward_extend(
        self,
        q: torch.Tensor,
        k: torch.Tensor,
        v: torch.Tensor,
        layer: RadixAttention,
        forward_batch: ForwardBatch,

View on GitHub (pinned to 0132848349)

Solutions

Run with --kv-cache-dtype bfloat16 (or fp8 disabled) when using hpc_ops on non-HunYuan models
Keep fp8_e4m3 KV cache but switch to an attention backend that supports fp8 KV without the fused op (e.g. flashinfer)
If you are porting a model, wire fused_qk_rope_store_kv_fp8 into its QKNorm+RoPE path so hpc_q_scale is populated

Example fix

# before
python -m sglang.launch_server --model Qwen2.5-72B --attention-backend hpc_ops --kv-cache-dtype fp8_e4m3
# after
python -m sglang.launch_server --model Qwen2.5-72B --attention-backend hpc_ops --kv-cache-dtype bfloat16

Defensive patterns

Strategy: validation

Validate before calling

def fp8_ok_with_hpc_ops(model_arch: str) -> bool:
    return model_arch in ('HunYuanV3ForCausalLM',)  # archs wired to fused_qk_rope_store_kv_fp8

if server_args.kv_cache_dtype == 'fp8_e4m3' and not fp8_ok_with_hpc_ops(model_arch):
    server_args.kv_cache_dtype = 'bfloat16'

Try / catch

try:
    out = backend.forward_extend(...)
except RuntimeError as e:
    if 'fused_qk_rope_store_kv_fp8' in str(e):
        restart_with_kv_dtype('bfloat16')
    else:
        raise

Prevention

Treat hpc_ops+fp8 KV as HunYuan-only
Validate server_args combos in a startup sanity-check script
Watch release notes for newly supported fp8 fused-op models

When it happens

Trigger: Running --attention-backend hpc_ops together with --kv-cache-dtype fp8_e4m3 on any model other than HunYuan V3 (or a model that does not call the fused_qk_rope_store_kv_fp8 op), so metadata.hpc_q_scale is None when attention runs.

Common situations: User enables fp8 KV cache to save memory and picks hpc_ops for speed on a non-HunYuan model; the combination is unsupported because no other model emits the required Q scales.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/995dcf930e370df5. Report an issue: GitHub.