sgl-project/sglang · error · TypeError

kv-canary: scatter_req_token_ids offsets must be int64, got

Error message

kv-canary: scatter_req_token_ids offsets must be int64, got {offsets.dtype}

What it means

The scatter launcher requires the CSR offsets vector to be torch.int64; offsets delimit token spans and are compared/indexed as 64-bit values in the Triton kernel. A different dtype raises TypeError before launch.

Source

Thrown at python/sglang/kernels/ops/kv_canary/scatter_req_token_ids.py:71

            f"{tuple(offsets.shape)}"
        )
    if req_pool_indices.dim() != 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be 1-D, got shape "
            f"{tuple(req_pool_indices.shape)}"
        )
    if pool_out.dim() != 2:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids pool_out must be 2-D, got shape "
            f"{tuple(pool_out.shape)}"
        )
    if flat_in.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids flat_in must be int64, got "
            f"{flat_in.dtype}"
        )
    if offsets.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids offsets must be int64, got "
            f"{offsets.dtype}"
        )
    if req_pool_indices.dtype != torch.int64:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids req_pool_indices must be int64, got "
            f"{req_pool_indices.dtype}"
        )
    if pool_out.dtype != torch.int32:
        raise TypeError(
            f"kv-canary: scatter_req_token_ids pool_out must be int32, got "
            f"{pool_out.dtype}"
        )

    bs = int(req_pool_indices.shape[0])
    if int(offsets.shape[0]) != bs + 1:
        raise ValueError(
            f"kv-canary: scatter_req_token_ids offsets length {offsets.shape[0]} != "

View on GitHub (pinned to 0132848349)

Solutions

Cast offsets to int64: offsets.to(torch.int64)
Allocate the offsets buffer with dtype=torch.int64 from the start

Example fix

# before
offsets = torch.cumsum(lens, 0).to(torch.int32)
# after
offsets = torch.cumsum(lens.to(torch.int64), 0)

Defensive patterns

Strategy: type-guard

Validate before calling

assert offsets.dtype == torch.int64, offsets.dtype

Type guard

def is_int64(t: torch.Tensor) -> bool:
    return t.dtype == torch.int64

Prevention

Build offsets in int64 from the start; converting int32 cumsums later invites overflow too

When it happens

Trigger: Building offsets with torch.int32 or torch.int16 cumsum results and passing them to launch_scatter_req_token_ids_kernel.

Common situations: Memory-optimized ragged metadata stored as int32 in the scheduler, fed directly to the kernel.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/09c4fbdbc615cfa6. Report an issue: GitHub.