{"record":{"id":"50c38797f41c65ae","repo":"sgl-project/sglang","slug":"attn-sink-must-be-a-cuda-tensor","errorCode":null,"errorMessage":"attn_sink must be a CUDA tensor","messagePattern":"attn_sink must be a CUDA tensor","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":413,"sourceCode":"                \"topk_length values must satisfy \" f\"0 <= topk_length <= topk ({topk})\"\n            )\n\n    if d_v != 512:\n        raise ValueError(\n            f\"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}\"\n        )\n\n    if attn_sink is not None and topk_length is None:\n        raise ValueError(\"attn_sink requires topk_length to be provided as well\")\n\n    if attn_sink is not None:\n        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:\n            raise ValueError(\n                f\"attn_sink must be float32 with shape ({h_q},), got \"\n                f\"{tuple(attn_sink.shape)}/{attn_sink.dtype}\"\n            )\n        if not attn_sink.is_cuda:\n            raise ValueError(\"attn_sink must be a CUDA tensor\")\n        if attn_sink.device != device:\n            raise ValueError(\n                f\"attn_sink must be on q's device {device}, got {attn_sink.device}\"\n            )\n        if not attn_sink.is_contiguous():\n            raise ValueError(\"attn_sink must be contiguous\")\n\n    for name, scale in ((\"q_scale\", q_scale), (\"kv_scale\", kv_scale)):\n        if not isinstance(scale, torch.Tensor):\n            raise ValueError(f\"{name} must be a torch.Tensor\")\n        if not scale.is_cuda:\n            raise ValueError(f\"{name} must be a CUDA tensor\")\n        if scale.device != device:\n            raise ValueError(\n                f\"{name} must be on q's device {device}, got {scale.device}\"\n            )\n        if scale.dtype != torch.float32:\n            raise ValueError(f\"{name} must be float32, got {scale.dtype}\")","sourceCodeStart":395,"sourceCodeEnd":431,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L395-L431","documentation":"The optional attn_sink argument to sparse_mla_q8kv8_prefill_fwd must live on CUDA memory. The kernel dereferences the sink pointer on the GPU, so a CPU tensor would cause illegal memory access; the Python validation rejects it up front.","triggerScenarios":"Calling sparse_mla_q8kv8_prefill_fwd(..., attn_sink=cpu_tensor) where cpu_tensor.is_cuda is False, e.g. a tensor created without a device= argument or loaded from checkpoint on CPU.","commonSituations":"Loading sink values from a safetensors checkpoint (which defaults to CPU) and passing them straight to the kernel without a .to(q.device) transfer.","solutions":["Move the sink to the compute device: attn_sink = attn_sink.to(q.device, non_blocking=True)","Allocate the sink directly on q.device when constructing it"],"exampleFix":"// before\nattn_sink = torch.zeros(h_q, dtype=torch.float32)  # CPU\n// after\nattn_sink = torch.zeros(h_q, dtype=torch.float32, device=q.device)","handlingStrategy":"validation","validationCode":"assert attn_sink.is_cuda, 'attn_sink must be on CUDA'","typeGuard":"def sink_on_cuda(t: torch.Tensor) -> bool:\n    return t.is_cuda","tryCatchPattern":null,"preventionTips":["Always allocate per-layer metadata tensors with device=q.device","After loading checkpoints, move small buffers to the model device in one pass"],"tags":["mla","sparse-attention","tensor-validation","cuda-device"],"backgroundTag":"tensor-on-wrong-device","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}