sgl-project/sglang · error · ValueError

kv-canary: RealKvSource.tensor dim-1 byte width must be a mu

Error message

kv-canary: RealKvSource.tensor dim-1 byte width must be a multiple of 16, got {row_stride_bytes} bytes (shape={tuple(self.tensor.shape)}, dtype={self.tensor.dtype})

What it means

Beyond rank, the kernel requires each row of RealKvSource.tensor to be a whole number of 16-byte units: dim-1 byte width (shape[1] * element_size) must be divisible by 16 for the vectorized 16-byte reads. __post_init__ raises ValueError with the computed row width otherwise.

Source

Thrown at python/sglang/kernels/ops/kv_canary/verify.py:118

                f"kv-canary: RealKvSource.num_bytes_per_token must be a positive multiple of 16, "
                f"got {self.num_bytes_per_token}"
            )
        if (
            self.read_bytes <= 0
            or self.read_bytes > self.num_bytes_per_token
            or self.read_bytes % 16 != 0
        ):
            raise ValueError(
                f"kv-canary: RealKvSource.read_bytes must be a positive multiple of 16 in "
                f"(0, num_bytes_per_token={self.num_bytes_per_token}], got {self.read_bytes}"
            )
        if self.tensor.ndim < 2:
            raise ValueError(
                f"kv-canary: RealKvSource.tensor must be at least 2-D, got shape {tuple(self.tensor.shape)}"
            )
        row_stride_bytes = int(self.tensor.shape[1]) * self.tensor.element_size()
        if row_stride_bytes % 16 != 0:
            raise ValueError(
                f"kv-canary: RealKvSource.tensor dim-1 byte width must be a multiple of 16, "
                f"got {row_stride_bytes} bytes (shape={tuple(self.tensor.shape)}, "
                f"dtype={self.tensor.dtype})"
            )


@dataclass(frozen=True, slots=True, kw_only=True)
class VerifyOrWriteContext:
    """Shared launch context for canary verify/write kernels.

    Fields:
        canary_buf: Canary buffer this launch verifies or writes, shape [num_slots, slot_stride_bytes], uint8.
            slot_stride_bytes is read from canary_buf.shape[1].
        kernel_kind: CanaryLaunchTag identifying which launch fired. Stamped (as int) into every violation row
            so host can attribute a violation back to its source launch.
        violation_ring: Global append-only sink, shape [ring_capacity, VIOLATION_FIELDS], int64. Shared across
            all canary launches; fill-once.
        violation_write_index: Global monotonic violation counter, shape [1], int32.

View on GitHub (pinned to 0132848349)

Solutions

1. Pad shape[1] (with zeros) so shape[1] * element_size % 16 == 0
2. Verify num_bytes_per_token matches shape[1] * element_size and adjust the view/reshape
3. Avoid slicing columns; pass the full row and limit reads via read_bytes instead

Example fix

# before
src = RealKvSource(tensor=kv[:, :7], ...)  # 14 bytes/row for fp16
# after
src = RealKvSource(tensor=kv, ...)  # full rows; use read_bytes to limit bytes read

row = tensor.shape[1] * tensor.element_size()
assert row % 16 == 0, f"row width {row} bytes not 16-aligned"

When it happens

Trigger: A tensor whose shape[1] * element_size is not a multiple of 16 — e.g. an odd number of fp16 elements per row (shape[1]=7 with 2-byte dtype gives 14 bytes), or a padded layout with a non-16 stride.

Common situations: Exotic head counts/head dims; column slicing that leaves a non-16-byte row width; viewing the cache with an incompatible dim-1 size.

Related errors

• kv-canary: RealKvSource.num_bytes_per_token must be a positi
• kv-canary: RealKvSource.read_bytes must be a positive multip
• v_cache must be provided
• kv-canary: {name} must be on {reference_name}'s device {refe
• kv-canary: launch_canary_plan_kernels_torch_reference verify

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c5844e7570dde6fd. Report an issue: GitHub.