sgl-project/sglang · error · ValueError

kv-canary: RealKvSource.num_bytes_per_token must be a positi

Error message

kv-canary: RealKvSource.num_bytes_per_token must be a positive multiple of 16, got {self.num_bytes_per_token}

What it means

RealKvSource.num_bytes_per_token must be a positive multiple of 16 because the canary kernel reads KV rows in 16-byte chunks (vectorized loads). __post_init__ raises ValueError for zero, negative, or non-multiple-of-16 values.

Source

Thrown at python/sglang/kernels/ops/kv_canary/verify.py:99

        page_size: Number of slots packed into one row of dim 0. ``>= 1``.
        num_bytes_per_token: Bytes per slot in the dim-1 strip the canary reads. Must be a positive
            multiple of 16.
        read_bytes: Leading bytes (out of ``num_bytes_per_token``) per slot folded into the fingerprint.
            Must be a positive multiple of 16, ``<= num_bytes_per_token``.
    """

    tensor: torch.Tensor
    page_size: int
    num_bytes_per_token: int
    read_bytes: int

    def __post_init__(self) -> None:
        if self.page_size < 1:
            raise ValueError(
                f"kv-canary: RealKvSource.page_size must be >= 1, got {self.page_size}"
            )
        if self.num_bytes_per_token <= 0 or self.num_bytes_per_token % 16 != 0:
            raise ValueError(
                f"kv-canary: RealKvSource.num_bytes_per_token must be a positive multiple of 16, "
                f"got {self.num_bytes_per_token}"
            )
        if (
            self.read_bytes <= 0
            or self.read_bytes > self.num_bytes_per_token
            or self.read_bytes % 16 != 0
        ):
            raise ValueError(
                f"kv-canary: RealKvSource.read_bytes must be a positive multiple of 16 in "
                f"(0, num_bytes_per_token={self.num_bytes_per_token}], got {self.read_bytes}"
            )
        if self.tensor.ndim < 2:
            raise ValueError(
                f"kv-canary: RealKvSource.tensor must be at least 2-D, got shape {tuple(self.tensor.shape)}"
            )
        row_stride_bytes = int(self.tensor.shape[1]) * self.tensor.element_size()
        if row_stride_bytes % 16 != 0:

View on GitHub (pinned to 0132848349)

Solutions

1. Compute num_bytes_per_token = num_heads * head_dim * tensor.element_size() and verify % 16 == 0
2. Round up to the next multiple of 16 if the layout has padding
3. Check for accidentally passing element count instead of byte count (missing element_size factor)

Example fix

# before
src = RealKvSource(num_bytes_per_token=num_heads * head_dim, ...)  # forgot bytes
# after
src = RealKvSource(num_bytes_per_token=num_heads * head_dim * kv.element_size(), ...)

nb = num_heads * head_dim * kv.element_size()
assert nb > 0 and nb % 16 == 0, nb

When it happens

Trigger: Constructing RealKvSource with num_bytes_per_token derived from head_dim * head_count * element_size that isn't a multiple of 16 (e.g. bf16 with an odd byte count), or passing 0.

Common situations: Switching model config to an unusual head_dim/dtype combination; forgetting to multiply by 2 bytes for fp16/bf16 when computing byte width.

Related errors

• kv-canary: RealKvSource.read_bytes must be a positive multip
• kv-canary: RealKvSource.tensor dim-1 byte width must be a mu
• v_cache must be provided
• kv-canary: {name} must be on {reference_name}'s device {refe
• kv-canary: launch_canary_plan_kernels_torch_reference verify

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7c3c0451087babf3. Report an issue: GitHub.