sgl-project/sglang · error · TypeError

kv-canary: scatter_req_token_ids pool_out must be int32, got

Error message

kv-canary: scatter_req_token_ids pool_out must be int32, got {pool_out.dtype}

What it means

Unlike the int64 inputs, pool_out must be torch.int32: the destination token-id pool is stored in 32-bit to halve memory, and the Triton kernel writes int32 words. A different dtype raises TypeError.

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:81

            f"{tuple(pool_out.shape)}"
        )
    if flat_in.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
            f"{flat_in.dtype}"
        )
    if offsets.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids offsets must be int64, got "
            f"{offsets.dtype}"
        )
    if req_pool_indices.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be int64, got "
            f"{req_pool_indices.dtype}"
        )
    if pool_out.dtype != torch.int32:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids pool_out must be int32, got "
            f"{pool_out.dtype}"
        )

    bs = int(req_pool_indices.shape[0])
    if int(offsets.shape[0]) != bs + 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets length {offsets.shape[0]} != "
            f"bs+1 ({bs + 1})"
        )
    if bs + 1 > _SCATTER_BATCH_BLOCK:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids bs+1={bs + 1} exceeds BATCH_BLOCK="
            f"{_SCATTER_BATCH_BLOCK}; bump _SCATTER_BATCH_BLOCK if real workloads need this"
        )

    num_tokens = int(flat_in.shape[0])
    if num_tokens == 0:

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate pool_out with dtype=torch.int32
2. Cast an existing buffer: pool_out.to(torch.int32) (prefer allocating correctly to avoid a copy)

Example fix

# before
pool_out = torch.empty((num_reqs, max_len), dtype=torch.int64)
# after
pool_out = torch.empty((num_reqs, max_len), dtype=torch.int32)

assert pool_out.dtype == torch.int32, pool_out.dtype

def is_int32(t: torch.Tensor) -> bool:
    return t.dtype == torch.int32

When it happens

Trigger: Allocating pool_out with dtype=torch.int64 (or defaulting from an int64 producer) and passing it to launch_scatter_req_token_ids_kernel.

Common situations: Copy-pasting the allocation of the int64 inputs for the output pool; unifying dtypes across a pipeline and forgetting pool_out is the exception.

Related errors

• kv-canary: scatter_req_token_ids flat_in must be int64, got
• kv-canary: scatter_req_token_ids offsets must be int64, got
• kv-canary: scatter_req_token_ids req_pool_indices must be in
• kv-canary: {name} must be on {reference_name}'s device {refe
• Unsupported dtype {k.dtype}. Supported: bfloat16, float16

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/77c238349aca21f2. Report an issue: GitHub.