sgl-project/sglang · error · TypeError
kv-canary: scatter_req_token_ids pool_out must be int32, got
Error message
kv-canary: scatter_req_token_ids pool_out must be int32, got {pool_out.dtype} What it means
kv-canary scatter kernel validation: the pool output tensor must have dtype torch.int32. Allocate pool_out as int32 or cast it.
Solutions
- Allocate pool_out with dtype=torch.int32
- Cast an existing buffer: pool_out.to(torch.int32) (prefer allocating correctly to avoid a copy)
Example fix
# before pool_out = torch.empty((num_reqs, max_len), dtype=torch.int64) # after pool_out = torch.empty((num_reqs, max_len), dtype=torch.int32)
Defensive patterns
Strategy: type-guard
Validate before calling
assert pool_out.dtype == torch.int32, pool_out.dtype
Type guard
def is_int32(t: torch.Tensor) -> bool:
return t.dtype == torch.int32 Prevention
- Remember the asymmetry: inputs int64, pool_out int32
When it happens
Trigger: Allocating pool_out with dtype=torch.int64 (or defaulting from an int64 producer) and passing it to launch_scatter_req_token_ids_kernel.
Common situations: Copy-pasting the allocation of the int64 inputs for the output pool; unifying dtypes across a pipeline and forgetting pool_out is the exception.
Related errors
- kv-canary: scatter_req_token_ids flat_in must be int64, got
- kv-canary: scatter_req_token_ids offsets must be int64, got
- kv-canary: scatter_req_token_ids req_pool_indices must be…
- kv-canary: must be on 's device , got
- Unsupported dtype . Supported: bfloat16, float16
AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28).
Data as JSON: /api/errors/77c238349aca21f2.
Report an issue: GitHub.
Appendix: source
Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:81
f"{tuple(pool_out.shape)}"
)
if flat_in.dtype != torch.int64:
raise TypeError(
f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
f"{flat_in.dtype}"
)
if offsets.dtype != torch.int64:
raise TypeError(
f"kv-canary: scatter_req_token_ids offsets must be int64, got "
f"{offsets.dtype}"
)
if req_pool_indices.dtype != torch.int64:
raise TypeError(
f"kv-canary: scatter_req_token_ids req_pool_indices must be int64, got "
f"{req_pool_indices.dtype}"
)
if pool_out.dtype != torch.int32:
raise TypeError(
f"kv-canary: scatter_req_token_ids pool_out must be int32, got "
f"{pool_out.dtype}"
)
bs = int(req_pool_indices.shape[0])
if int(offsets.shape[0]) != bs + 1:
raise ValueError(
f"kv-canary: scatter_req_token_ids offsets length {offsets.shape[0]} != "
f"bs+1 ({bs + 1})"
)
if bs + 1 > _SCATTER_BATCH_BLOCK:
raise ValueError(
f"kv-canary: scatter_req_token_ids bs+1={bs + 1} exceeds BATCH_BLOCK="
f"{_SCATTER_BATCH_BLOCK}; bump _SCATTER_BATCH_BLOCK if real workloads need this"
)
num_tokens = int(flat_in.shape[0])
if num_tokens == 0:View on GitHub (pinned to 0132848349)