{"record":{"id":"317d43312fd6608e","repo":"sgl-project/sglang","slug":"name-must-be-a-torch-tensor","errorCode":null,"errorMessage":"{name} must be a torch.Tensor","messagePattern":"(.+?) must be a torch\\.Tensor","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":423,"sourceCode":"\n    if attn_sink is not None:\n        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:\n            raise ValueError(\n                f\"attn_sink must be float32 with shape ({h_q},), got \"\n                f\"{tuple(attn_sink.shape)}/{attn_sink.dtype}\"\n            )\n        if not attn_sink.is_cuda:\n            raise ValueError(\"attn_sink must be a CUDA tensor\")\n        if attn_sink.device != device:\n            raise ValueError(\n                f\"attn_sink must be on q's device {device}, got {attn_sink.device}\"\n            )\n        if not attn_sink.is_contiguous():\n            raise ValueError(\"attn_sink must be contiguous\")\n\n    for name, scale in ((\"q_scale\", q_scale), (\"kv_scale\", kv_scale)):\n        if not isinstance(scale, torch.Tensor):\n            raise ValueError(f\"{name} must be a torch.Tensor\")\n        if not scale.is_cuda:\n            raise ValueError(f\"{name} must be a CUDA tensor\")\n        if scale.device != device:\n            raise ValueError(\n                f\"{name} must be on q's device {device}, got {scale.device}\"\n            )\n        if scale.dtype != torch.float32:\n            raise ValueError(f\"{name} must be float32, got {scale.dtype}\")\n        if scale.numel() != 1:\n            raise ValueError(\n                f\"{name} must be a scalar tensor, got shape {tuple(scale.shape)}\"\n            )\n        if not scale.is_contiguous():\n            raise ValueError(f\"{name} must be contiguous\")\n\n    if out is None:\n        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)\n    else:","sourceCodeStart":405,"sourceCodeEnd":441,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L405-L441","documentation":"sparse_mla_q8kv8_prefill_fwd validates q_scale and kv_scale in a loop; both must be torch.Tensor objects (not Python floats) because the kernel dereferences them as GPU buffers holding the quantization de-scale factors.","triggerScenarios":"Calling the function with q_scale=1.0 or kv_scale=np.float32(0.5) — any non-tensor scalar (int, float, numpy scalar) triggers it.","commonSituations":"Porting code from an API that accepted float scales, or passing raw fp8 de-scale amplitudes computed in Python instead of materialized tensors.","solutions":["Wrap scalars: q_scale = torch.tensor(value, dtype=torch.float32, device=q.device)","If scales come from a quantization config, precompute them as 1-element CUDA tensors once at setup"],"exampleFix":"// before\nsparse_mla_q8kv8_prefill_fwd(q, k, v, ..., q_scale=1.0, kv_scale=0.5)\n// after\ndev = q.device\nq_scale = torch.tensor(1.0, dtype=torch.float32, device=dev)\nkv_scale = torch.tensor(0.5, dtype=torch.float32, device=dev)\nsparse_mla_q8kv8_prefill_fwd(q, k, v, ..., q_scale=q_scale, kv_scale=kv_scale)","handlingStrategy":"type-guard","validationCode":"if not isinstance(q_scale, torch.Tensor):\n    q_scale = torch.tensor(q_scale, dtype=torch.float32, device=q.device)","typeGuard":"def as_scale_tensor(s, device) -> torch.Tensor:\n    if not isinstance(s, torch.Tensor):\n        s = torch.tensor(s, dtype=torch.float32, device=device)\n    return s","tryCatchPattern":null,"preventionTips":["Materialize quantization scales as 1-element tensors once at model setup","Never pass raw Python/numpy scalars to kernel wrappers"],"tags":["mla","quantization","scale-validation","tensor-validation"],"backgroundTag":"tensor-shape-validation-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}