{"record":{"id":"78f02a647385ab76","repo":"sgl-project/sglang","slug":"q-must-be-torch-float8-e4m3fn-got-q-dtype","errorCode":null,"errorMessage":"q must be torch.float8_e4m3fn, got {q.dtype}","messagePattern":"q must be torch\\.float8_e4m3fn, got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":331,"sourceCode":"\n    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all\n    # accesses on q's CUDA device. Reject contract violations before launch.\n    if not q.is_cuda:\n        raise ValueError(\"q must be a CUDA tensor\")\n    if not kv.is_cuda:\n        raise ValueError(\"kv must be a CUDA tensor\")\n    if not indices.is_cuda:\n        raise ValueError(\"indices must be a CUDA tensor\")\n\n    if kv.device != device:\n        raise ValueError(f\"kv must be on q's device {device}, got {kv.device}\")\n    if indices.device != device:\n        raise ValueError(\n            f\"indices must be on q's device {device}, got {indices.device}\"\n        )\n\n    if q.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"q must be torch.float8_e4m3fn, got {q.dtype}\")\n    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():\n        raise ValueError(\"kv must be contiguous\")\n    if not indices.is_contiguous():\n        raise ValueError(\"indices must be contiguous\")\n\n    if kv_d_qk != d_qk:\n        raise ValueError(f\"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}\")\n\n    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.\n    # Reject unpadded TP-local head counts instead of launching zero CTAs and\n    # returning uninitialized outputs, which can appear to callers as a hang or\n    # a later collective failure.\n    if h_q == 0 or h_q % 64 != 0:","sourceCodeStart":313,"sourceCodeEnd":349,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L313-L349","documentation":"The sparse Q8KV8 prefill kernel (SM90) quantizes and computes in FP8, so the query tensor q must be torch.float8_e4m3fn. Passing bf16/fp16/fp32 queries is rejected because the kernel has no dequant/quant path for q.","triggerScenarios":"Calling sparse_mla_q8kv8_prefill_fwd with a q tensor in torch.bfloat16 (e.g. reusing the decode-path q tensor that was never quantized to FP8).","commonSituations":"Wiring a new model/attention backend where q is produced in the default model dtype (bf16) and the FP8 quantization step (per-token or static scale cast to float8_e4m3fn) was skipped; version changes that moved quantization into a separate op.","solutions":["Quantize q to float8_e4m3fn with the same scale scheme used for kv before calling (e.g. q.to(torch.float8_e4m3fn) or the framework's fp8 quantize helper)","Ensure your backend's q8kv8 sparse path actually enables FP8 quantization of q, not just kv","If you only have bf16 tensors, use the non-Q8 non-sparse prefill path instead of this kernel"],"exampleFix":"// before\nq_fp8 = q  # still bfloat16\nout = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_fp8, indices)\n// after\nq_fp8 = q.to(torch.float8_e4m3fn)\nout = sparse_mla_q8kv8_prefill_fwd(q_fp8, kv_fp8, indices)","handlingStrategy":"validation","validationCode":"if q.dtype != torch.float8_e4m3fn: q = q.to(torch.float8_e4m3fn)","typeGuard":"def is_fp8_e4m3(t: torch.Tensor) -> bool:\n    return t.dtype == torch.float8_e4m3fn","tryCatchPattern":null,"preventionTips":["Centralize FP8 casting in one helper used by all q8kv8 call sites","Add unit assertions on dtype in your backend's forward"],"tags":["dtype","fp8","sparse-attention","sglang"],"backgroundTag":"tensor-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}