sgl-project/sglang · error · ValueError

kv-canary: scatter_req_token_ids offsets must be 1-D, got sh

Error message

kv-canary: scatter_req_token_ids offsets must be 1-D, got shape {tuple(offsets.shape)}

What it means

The scatter launcher requires offsets to be 1-D: it is the CSR-style prefix-offset vector of length bs+1 delimiting each request's token span in flat_in. Passing a 2-D/scalar offsets tensor raises ValueError before kernel launch.

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:51

        flat_in: ``[total_tokens]`` int64 device tensor of objects, flattened
            per-req in req order.
        offsets: ``[bs + 1]`` int64 device tensor (host-computed cumsum of per-req
            lengths). ``offsets[bs] == total_tokens``.
        req_pool_indices: ``[bs]`` int64 device tensor of pool row indices.
        pool_out: ``[max_reqs, max_context_len]`` int32 device tensor of objects.
            Mutated in-place; rows not addressed by ``req_pool_indices`` are untouched.

    Implementation notes:
        - Linear scan over ``offsets`` (``BATCH_BLOCK >= bs + 1``); fits easily in
          registers for the workloads kv-canary handles (``bs <= a few thousand``).
    """
    if flat_in.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape "
            f"{tuple(flat_in.shape)}"
        )
    if offsets.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets must be 1-D, got shape "
            f"{tuple(offsets.shape)}"
        )
    if req_pool_indices.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape "
            f"{tuple(req_pool_indices.shape)}"
        )
    if pool_out.dim() != 2:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape "
            f"{tuple(pool_out.shape)}"
        )
    if flat_in.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
            f"{flat_in.dtype}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Squeeze the tensor: offsets = offsets.squeeze(0) or .reshape(-1)
2. Build offsets with 1-D cumsum: torch.zeros(bs+1, dtype=torch.int64) then fill
3. Add an assertion offsets.dim() == 1 right after construction in the caller

Example fix

# before
offsets = torch.cumsum(lens, 0, keepdim=True)  # wrong dims
# after
offsets = torch.zeros(bs + 1, dtype=torch.int64)
offsets[1:] = torch.cumsum(lens, 0)

assert offsets.dim() == 1, offsets.shape

def is_1d(t: torch.Tensor) -> bool:
    return t.dim() == 1

When it happens

Trigger: Passing offsets with an extra dimension (e.g. shape [1, bs+1] after a keepdim operation) or a 0-D scalar to launch_scatter_req_token_ids_kernel.

Common situations: Offsets produced by torch.cumsum(..., keepdim=True) or slicing a 2-D buffer; bugs in ragged-batch construction.

Related errors

• kv-canary: scatter_req_token_ids flat_in must be 1-D, got sh
• kv-canary: scatter_req_token_ids req_pool_indices must be 1-
• kv-canary: scatter_req_token_ids pool_out must be 2-D, got s
• kv-canary: RealKvSource.tensor must be at least 2-D, got sha
• rope_pool_fused expects pool tensors to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/613f3333e7239e01. Report an issue: GitHub.