sgl-project/sglang · error · ValueError

kv-canary: scatter_req_token_ids pool_out must be 2-D, got s

Error message

kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape {tuple(pool_out.shape)}

What it means

The scatter launcher requires pool_out — the [num_reqs, max_context_len] destination pool being filled with token ids — to be exactly 2-D. Any other rank raises ValueError since the kernel indexes pool_out[req, pos].

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:61

          registers for the workloads kv-canary handles (``bs <= a few thousand``).
    """
    if flat_in.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape "
            f"{tuple(flat_in.shape)}"
        )
    if offsets.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets must be 1-D, got shape "
            f"{tuple(offsets.shape)}"
        )
    if req_pool_indices.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape "
            f"{tuple(req_pool_indices.shape)}"
        )
    if pool_out.dim() != 2:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape "
            f"{tuple(pool_out.shape)}"
        )
    if flat_in.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
            f"{flat_in.dtype}"
        )
    if offsets.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids offsets must be int64, got "
            f"{offsets.dtype}"
        )
    if req_pool_indices.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be int64, got "
            f"{req_pool_indices.dtype}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Index down to 2-D: pool_out = token_pool[layer] or .squeeze(extra_dim)
2. Allocate the destination as torch.empty((num_reqs, max_context_len), dtype=torch.int32)

Example fix

# before
launch_scatter(..., pool_out=token_pool_3d)
# after
launch_scatter(..., pool_out=token_pool_3d[0])  # [num_reqs, max_context_len]

assert pool_out.dim() == 2, pool_out.shape

def is_2d(t: torch.Tensor) -> bool:
    return t.dim() == 2

When it happens

Trigger: Passing a 1-D flattened pool, a 3-D [layers, reqs, len] pool, or a pool view with an extra singleton dim to launch_scatter_req_token_ids_kernel.

Common situations: Slicing a multi-layer KV metadata buffer and forgetting to select the layer dim; passing the whole stack instead of pool[layer].

Related errors

• kv-canary: scatter_req_token_ids flat_in must be 1-D, got sh
• kv-canary: scatter_req_token_ids offsets must be 1-D, got sh
• kv-canary: scatter_req_token_ids req_pool_indices must be 1-
• kv-canary: RealKvSource.tensor must be at least 2-D, got sha
• rope_pool_fused expects pool tensors to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b1841d813ee096bc. Report an issue: GitHub.