{"record":{"id":"f1f5ada3bbfdb861","repo":"sgl-project/sglang","slug":"kv-must-be-torch-float8-e4m3fn-got-kv-dtype","errorCode":null,"errorMessage":"kv must be torch.float8_e4m3fn, got {kv.dtype}","messagePattern":"kv must be torch\\.float8_e4m3fn, got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":333,"sourceCode":"    # accesses on q's CUDA device. Reject contract violations before launch.\n    if not q.is_cuda:\n        raise ValueError(\"q must be a CUDA tensor\")\n    if not kv.is_cuda:\n        raise ValueError(\"kv must be a CUDA tensor\")\n    if not indices.is_cuda:\n        raise ValueError(\"indices must be a CUDA tensor\")\n\n    if kv.device != device:\n        raise ValueError(f\"kv must be on q's device {device}, got {kv.device}\")\n    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():\n        raise ValueError(\"kv must be contiguous\")\n    if not indices.is_contiguous():\n        raise ValueError(\"indices must be contiguous\")\n\n    if kv_d_qk != d_qk:\n        raise ValueError(f\"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}\")\n\n    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.\n    # Reject unpadded TP-local head counts instead of launching zero CTAs and\n    # returning uninitialized outputs, which can appear to callers as a hang or\n    # a later collective failure.\n    if h_q == 0 or h_q % 64 != 0:\n        raise ValueError(\n            \"sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive \"","sourceCodeStart":315,"sourceCodeEnd":351,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L315-L351","documentation":"The key/value cache tensor kv must be torch.float8_e4m3fn for the Q8KV8 sparse prefill kernel, which reads KV directly as FP8 with e4m3 layout. Any other dtype (e.g. bf16, e5m2) is rejected up front to avoid garbage results or a kernel crash.","triggerScenarios":"Passing a bf16 KV cache or an FP8 cache in torch.float8_e5m2 format to sparse_mla_q8kv8_prefill_fwd.","commonSituations":"The KV cache pool was allocated with the model dtype instead of the FP8 q8kv8 dtype; the e4m3 vs e5m2 FP8 variant was configured incorrectly (SGLANG FP8 format settings); mixing decode-path bf16 cache with the new sparse prefill kernel.","solutions":["Ensure the KV cache tensor for this path is stored as torch.float8_e4m3fn (allocate/convert the cache accordingly)","Check your FP8 configuration selects e4m3, not e5m2","If you cannot use an FP8 cache, route prefill through the non-Q8 backend"],"exampleFix":"// before\nout = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_bf16, indices)\n// after\nkv_fp8 = kv_bf16.to(torch.float8_e4m3fn)\nout = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_fp8, indices)","handlingStrategy":"validation","validationCode":"assert kv.dtype == torch.float8_e4m3fn, f\"kv dtype {kv.dtype}\"","typeGuard":"def is_q8kv8_cache(kv: torch.Tensor) -> bool:\n    return kv.dtype == torch.float8_e4m3fn and kv.is_contiguous()","tryCatchPattern":null,"preventionTips":["Allocate the KV cache pool with torch.float8_e4m3fn when enabling q8kv8","Fail fast at server startup if cache dtype != e4m3 for this backend"],"tags":["dtype","fp8","kv-cache","sparse-attention"],"backgroundTag":"tensor-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}