sgl-project/sglang · error · ValueError

kv-canary: scatter_req_token_ids req_pool_indices must be 1-

Error message

kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape {tuple(req_pool_indices.shape)}

What it means

The scatter launcher requires req_pool_indices (per-request request-pool row indices) to be a 1-D tensor of length bs. A higher-rank tensor raises ValueError because the kernel indexes it by batch row.

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:56

        pool_out: ``[max_reqs, max_context_len]`` int32 device tensor of objects.
            Mutated in-place; rows not addressed by ``req_pool_indices`` are untouched.

    Implementation notes:
        - Linear scan over ``offsets`` (``BATCH_BLOCK >= bs + 1``); fits easily in
          registers for the workloads kv-canary handles (``bs <= a few thousand``).
    """
    if flat_in.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape "
            f"{tuple(flat_in.shape)}"
        )
    if offsets.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets must be 1-D, got shape "
            f"{tuple(offsets.shape)}"
        )
    if req_pool_indices.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape "
            f"{tuple(req_pool_indices.shape)}"
        )
    if pool_out.dim() != 2:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape "
            f"{tuple(pool_out.shape)}"
        )
    if flat_in.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
            f"{flat_in.dtype}"
        )
    if offsets.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids offsets must be int64, got "
            f"{offsets.dtype}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Apply .squeeze(-1) or .reshape(-1) to restore 1-D
2. Keep a canonical 1-D int64 req_pool_indices in the batch state and only reshape at other call sites

Example fix

# before
launch_scatter(..., req_pool_indices=rp.unsqueeze(-1))
# after
launch_scatter(..., req_pool_indices=rp)

assert req_pool_indices.dim() == 1, req_pool_indices.shape

def is_1d(t: torch.Tensor) -> bool:
    return t.dim() == 1

When it happens

Trigger: Passing req_pool_indices of shape [bs, 1] or [1, bs], e.g. after unsqueezing for another kernel's grid, to launch_scatter_req_token_ids_kernel.

Common situations: Reusing tensors shaped for a different kernel ABI; batching code that adds a dummy dimension.

Related errors

• kv-canary: scatter_req_token_ids flat_in must be 1-D, got sh
• kv-canary: scatter_req_token_ids offsets must be 1-D, got sh
• kv-canary: scatter_req_token_ids pool_out must be 2-D, got s
• kv-canary: RealKvSource.tensor must be at least 2-D, got sha
• rope_pool_fused expects pool tensors to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9f21eed65352a437. Report an issue: GitHub.