sgl-project/sglang · error · TypeError

kv-canary: scatter_req_token_ids flat_in must be int64, got

Error message

kv-canary: scatter_req_token_ids flat_in must be int64, got {flat_in.dtype}

What it means

The scatter kernel reads raw int64 words from flat_in (token ids with sentinel semantics encoded in 64-bit), so the launcher enforces dtype torch.int64 and raises TypeError otherwise. int32 inputs would be misread as pairs, corrupting the scatter.

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:66

            f"{tuple(flat_in.shape)}"
        )
    if offsets.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets must be 1-D, got shape "
            f"{tuple(offsets.shape)}"
        )
    if req_pool_indices.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape "
            f"{tuple(req_pool_indices.shape)}"
        )
    if pool_out.dim() != 2:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape "
            f"{tuple(pool_out.shape)}"
        )
    if flat_in.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
            f"{flat_in.dtype}"
        )
    if offsets.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids offsets must be int64, got "
            f"{offsets.dtype}"
        )
    if req_pool_indices.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be int64, got "
            f"{req_pool_indices.dtype}"
        )
    if pool_out.dtype != torch.int32:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids pool_out must be int32, got "
            f"{pool_out.dtype}"
        )

View on GitHub (pinned to 0132848349)

Solutions

  1. Cast: flat_in = flat_in.to(torch.int64) at the call site
  2. Ensure the producer of flat_in allocates with dtype=torch.int64

Example fix

# before
launch_scatter(..., flat_in=flat_in_int32)
# after
launch_scatter(..., flat_in=flat_in_int32.to(torch.int64))
Defensive patterns

Strategy: type-guard

Validate before calling

assert flat_in.dtype == torch.int64, flat_in.dtype

Type guard

def is_int64(t: torch.Tensor) -> bool:
    return t.dtype == torch.int64

Prevention

When it happens

Trigger: Calling launch_scatter_req_token_ids_kernel with flat_in of dtype torch.int32, torch.uint8, etc.

Common situations: Downstream code that stores token ids as int32 for memory efficiency; tensors coming from a tokenizer configured with a narrower dtype.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c0a0570391ff09ac. Report an issue: GitHub.