sgl-project/sglang · error · ValueError

kv-canary: scatter_req_token_ids flat_in must be 1-D, got sh

Error message

kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape {tuple(flat_in.shape)}

What it means

The scatter_req_token_ids kernel launcher validates that flat_in (the flattened token-id input) is a 1-D tensor; anything else (2-D batched, 3-D) is rejected with ValueError because the Triton kernel indexes it linearly over tokens.

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:46

    - ``rp = req_pool_indices[r]``
    - if ``pos < pool_max_context_len``:
      ``pool_out[rp, pos] = flat_in[t].to(int32)``

    Args:
        flat_in: ``[total_tokens]`` int64 device tensor of objects, flattened
            per-req in req order.
        offsets: ``[bs + 1]`` int64 device tensor (host-computed cumsum of per-req
            lengths). ``offsets[bs] == total_tokens``.
        req_pool_indices: ``[bs]`` int64 device tensor of pool row indices.
        pool_out: ``[max_reqs, max_context_len]`` int32 device tensor of objects.
            Mutated in-place; rows not addressed by ``req_pool_indices`` are untouched.

    Implementation notes:
        - Linear scan over ``offsets`` (``BATCH_BLOCK >= bs + 1``); fits easily in
          registers for the workloads kv-canary handles (``bs <= a few thousand``).
    """
    if flat_in.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids flat_in must be 1-D, got shape "
            f"{tuple(flat_in.shape)}"
        )
    if offsets.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets must be 1-D, got shape "
            f"{tuple(offsets.shape)}"
        )
    if req_pool_indices.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape "
            f"{tuple(req_pool_indices.shape)}"
        )
    if pool_out.dim() != 2:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape "
            f"{tuple(pool_out.shape)}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Flatten the input: flat_in = token_ids.reshape(-1)
2. Verify your offsets array was built against the same flattened layout
3. Check for accidental unsqueeze/add of a leading dim in upstream code

Example fix

# before
launch_scatter(..., flat_in=batch_token_ids)  # [bs, max_len]
# after
launch_scatter(..., flat_in=batch_token_ids.reshape(-1))

assert flat_in.dim() == 1, flat_in.shape

def is_flat_1d(t: torch.Tensor) -> bool:
    return t.dim() == 1

When it happens

Trigger: Calling launch_scatter_req_token_ids_kernel with a 2-D [bs, max_len] token tensor instead of the flattened [num_tokens] form, or forgetting .view(-1)/.flatten() after concatenation.

Common situations: Feeding the scheduler's batched token buffer directly instead of the ragged flat token stream; reshaping bugs after slicing.

Related errors

• kv-canary: scatter_req_token_ids offsets must be 1-D, got sh
• kv-canary: scatter_req_token_ids req_pool_indices must be 1-
• kv-canary: scatter_req_token_ids pool_out must be 2-D, got s
• kv-canary: RealKvSource.tensor must be at least 2-D, got sha
• rope_pool_fused expects pool tensors to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/23733157ef189951. Report an issue: GitHub.