{"record":{"id":"23733157ef189951","repo":"sgl-project/sglang","slug":"kv-canary-scatter-req-token-ids-flat-in-must-be-1","errorCode":null,"errorMessage":"kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape {tuple(flat_in.shape)}","messagePattern":"kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py","lineNumber":46,"sourceCode":"    - ``rp = req_pool_indices[r]``\n    - if ``pos < pool_max_context_len``:\n      ``pool_out[rp, pos] = flat_in[t].to(int32)``\n\n    Args:\n        flat_in: ``[total_tokens]`` int64 device tensor of objects, flattened\n            per-req in req order.\n        offsets: ``[bs + 1]`` int64 device tensor (host-computed cumsum of per-req\n            lengths). ``offsets[bs] == total_tokens``.\n        req_pool_indices: ``[bs]`` int64 device tensor of pool row indices.\n        pool_out: ``[max_reqs, max_context_len]`` int32 device tensor of objects.\n            Mutated in-place; rows not addressed by ``req_pool_indices`` are untouched.\n\n    Implementation notes:\n        - Linear scan over ``offsets`` (``BATCH_BLOCK >= bs + 1``); fits easily in\n          registers for the workloads kv-canary handles (``bs <= a few thousand``).\n    \"\"\"\n    if flat_in.dim() != 1:\n        raise ValueError(\n            f\"kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape \"\n            f\"{tuple(flat_in.shape)}\"\n        )\n    if offsets.dim() != 1:\n        raise ValueError(\n            f\"kv-canary: scatter_req_token_ids offsets must be 1-D, got shape \"\n            f\"{tuple(offsets.shape)}\"\n        )\n    if req_pool_indices.dim() != 1:\n        raise ValueError(\n            f\"kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape \"\n            f\"{tuple(req_pool_indices.shape)}\"\n        )\n    if pool_out.dim() != 2:\n        raise ValueError(\n            f\"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape \"\n            f\"{tuple(pool_out.shape)}\"\n        )","sourceCodeStart":28,"sourceCodeEnd":64,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py#L28-L64","documentation":"The scatter_req_token_ids kernel launcher validates that flat_in (the flattened token-id input) is a 1-D tensor; anything else (2-D batched, 3-D) is rejected with ValueError because the Triton kernel indexes it linearly over tokens.","triggerScenarios":"Calling launch_scatter_req_token_ids_kernel with a 2-D [bs, max_len] token tensor instead of the flattened [num_tokens] form, or forgetting .view(-1)/.flatten() after concatenation.","commonSituations":"Feeding the scheduler's batched token buffer directly instead of the ragged flat token stream; reshaping bugs after slicing.","solutions":["Flatten the input: flat_in = token_ids.reshape(-1)","Verify your offsets array was built against the same flattened layout","Check for accidental unsqueeze/add of a leading dim in upstream code"],"exampleFix":"# before\nlaunch_scatter(..., flat_in=batch_token_ids)  # [bs, max_len]\n# after\nlaunch_scatter(..., flat_in=batch_token_ids.reshape(-1))","handlingStrategy":"type-guard","validationCode":"assert flat_in.dim() == 1, flat_in.shape","typeGuard":"def is_flat_1d(t: torch.Tensor) -> bool:\n    return t.dim() == 1","tryCatchPattern":null,"preventionTips":["Always .reshape(-1) ragged token streams before scatter kernels"],"tags":["kv-cache","shape-validation","tensor-rank"],"backgroundTag":"tensor-shape-validation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}