sgl-project/sglang · error · ValueError

kv-canary: real_kv_sources[{i}].tensor (viewed as uint8) mus

Error message

kv-canary: real_kv_sources[{i}].tensor (viewed as uint8) must be 2-D, got {source_u8.dim()}-D

What it means

Each RealKvSource.tensor is reinterpreted as a uint8 byte buffer and must be a 2-D [slots, bytes_per_slot] layout because the CUDA ABI writes per-row offsets against a 2-D stride. A 1-D flat buffer or a 3-D tensor cannot be described by the single row-stride field in the ABI params.

Source

Thrown at python/sglang/kernels/ops/kv_canary/verify.py:386

def _build_real_kv_source_abi(
    *,
    real_kv_sources: tuple[RealKvSource, ...],
    device: torch.device,
) -> tuple[list[torch.Tensor], torch.Tensor]:
    padded_bufs: list[torch.Tensor] = []
    params = torch.zeros(
        (consts.MAX_REAL_KV_SOURCES, consts.REAL_KV_SOURCE_FIELDS_PER_ENTRY),
        dtype=torch.int32,
        device="cpu",
    )

    for i, source in enumerate(real_kv_sources):
        _assert_contiguous(source.tensor, f"real_kv_sources[{i}].tensor")
        source_u8 = source.tensor.view(torch.uint8)
        if source_u8.dim() != 2:
            raise ValueError(
                f"kv-canary: real_kv_sources[{i}].tensor (viewed as uint8) must be 2-D, "
                f"got {source_u8.dim()}-D"
            )
        padded_bufs.append(source_u8)
        params[i, consts.REAL_KV_SOURCE_FIELD_PAGE_SIZE] = source.page_size
        params[i, consts.REAL_KV_SOURCE_FIELD_NUM_BYTES_PER_TOKEN] = (
            source.num_bytes_per_token
        )
        params[i, consts.REAL_KV_SOURCE_FIELD_READ_BYTES] = source.read_bytes

    # Pad bufs (never read by the kernel — num_sources bounds the iteration); params already zero.
    dummy = torch.empty((1, 1), dtype=torch.uint8, device=device)
    for _ in range(len(real_kv_sources), consts.MAX_REAL_KV_SOURCES):
        padded_bufs.append(dummy)

    return padded_bufs, params

View on GitHub (pinned to 0132848349)

Solutions

1. Reshape the tensor to 2-D before passing: tensor.reshape(num_slots, bytes_per_slot) (with .contiguous() so the uint8 view is valid)
2. If the source is [pages, tokens, dim], flatten the trailing dims: t.view(t.size(0), -1)
3. Ensure the last dim's stride in bytes matches the page/slot size expected by the kernel params (page_size, num_bytes_per_token)

Example fix

# before
src = RealKvSource(tensor=flat_bytes_1d, page_size=16, num_bytes_per_token=...)
# after
src = RealKvSource(tensor=flat_bytes_1d.view(num_slots, slot_bytes), page_size=16, num_bytes_per_token=...)

u8 = src.tensor.view(torch.uint8)
assert u8.dim() == 2, f"source tensor must be 2-D as uint8, got {u8.dim()}-D"

def is_valid_kv_source(src) -> bool:
    return src.tensor.is_contiguous() and src.tensor.view(torch.uint8).dim() == 2

When it happens

Trigger: Passing a RealKvSource whose .tensor is 1-D (flattened bytes) or 3-D+ (e.g. [num_pages, page_size, head_dim]) to launch_canary_verify_kernel or launch_canary_write_kernel.

Common situations: Feeding a raw flat cache allocation, or an unflattened [pages, tokens, dim] KV tensor from a paged cache, directly as a source without reshaping to [slots, slot_bytes].

Related errors

• NPU packed attention requires q, k, and v in [T, N, D] layou
• The pointers must be multiple of 16 bytes.
• The last dimension ({input.shape[-1]}) x itemsize ({input.dt
• rope_pool_fused expects q/k/v to be 3-D
• rope_pool_fused expects positions/slots to be 1-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/13afa1628c9d95ab. Report an issue: GitHub.