sgl-project/sglang · error · TypeError

kv-canary: scatter_req_token_ids req_pool_indices must be in

Error message

kv-canary: scatter_req_token_ids req_pool_indices must be int64, got {req_pool_indices.dtype}

What it means

The scatter launcher requires req_pool_indices to be torch.int64, matching the request-pool index ABI used across kv-canary kernels. Any other dtype raises TypeError.

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:76

            f"{tuple(req_pool_indices.shape)}"
        )
    if pool_out.dim() != 2:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape "
            f"{tuple(pool_out.shape)}"
        )
    if flat_in.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
            f"{flat_in.dtype}"
        )
    if offsets.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids offsets must be int64, got "
            f"{offsets.dtype}"
        )
    if req_pool_indices.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be int64, got "
            f"{req_pool_indices.dtype}"
        )
    if pool_out.dtype != torch.int32:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids pool_out must be int32, got "
            f"{pool_out.dtype}"
        )

    bs = int(req_pool_indices.shape[0])
    if int(offsets.shape[0]) != bs + 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets length {offsets.shape[0]} != "
            f"bs+1 ({bs + 1})"
        )
    if bs + 1 > _SCATTER_BATCH_BLOCK:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids bs+1={bs + 1} exceeds BATCH_BLOCK="

View on GitHub (pinned to 0132848349)

Solutions

Cast: req_pool_indices.to(torch.int64)
Keep one canonical int64 copy of req_pool_indices for kv-canary calls

Example fix

# before
launch_scatter(..., req_pool_indices=rp_int32)
# after
launch_scatter(..., req_pool_indices=rp_int32.to(torch.int64))

Defensive patterns

Strategy: type-guard

Validate before calling

assert req_pool_indices.dtype == torch.int64, req_pool_indices.dtype

Type guard

def is_int64(t: torch.Tensor) -> bool:
    return t.dtype == torch.int64

Prevention

Cache an int64 view/copy of the request pool indices for kv-canary calls

When it happens

Trigger: Passing int32 request-pool indices (common when the allocator pool is int32) to launch_scatter_req_token_ids_kernel.

Common situations: Scheduler request-pool tensors stored as int32 for compactness; converting from numpy arrays that default to int32 on Windows.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fad7531a3148581a. Report an issue: GitHub.