sgl-project/sglang · error · ValueError

kv-canary: RealKvSource.read_bytes must be a positive multip

Error message

kv-canary: RealKvSource.read_bytes must be a positive multiple of 16 in (0, num_bytes_per_token={self.num_bytes_per_token}], got {self.read_bytes}

What it means

RealKvSource.read_bytes (how many bytes per token the canary actually fingerprints) must be a positive multiple of 16 and at most num_bytes_per_token. __post_init__ raises ValueError otherwise, since the kernel reads in 16-byte units and cannot read past a row.

Source

Thrown at python/sglang/kernels/ops/kv_canary/verify.py:108

    num_bytes_per_token: int
    read_bytes: int

    def __post_init__(self) -> None:
        if self.page_size < 1:
            raise ValueError(
                f"kv-canary: RealKvSource.page_size must be >= 1, got {self.page_size}"
            )
        if self.num_bytes_per_token <= 0 or self.num_bytes_per_token % 16 != 0:
            raise ValueError(
                f"kv-canary: RealKvSource.num_bytes_per_token must be a positive multiple of 16, "
                f"got {self.num_bytes_per_token}"
            )
        if (
            self.read_bytes <= 0
            or self.read_bytes > self.num_bytes_per_token
            or self.read_bytes % 16 != 0
        ):
            raise ValueError(
                f"kv-canary: RealKvSource.read_bytes must be a positive multiple of 16 in "
                f"(0, num_bytes_per_token={self.num_bytes_per_token}], got {self.read_bytes}"
            )
        if self.tensor.ndim < 2:
            raise ValueError(
                f"kv-canary: RealKvSource.tensor must be at least 2-D, got shape {tuple(self.tensor.shape)}"
            )
        row_stride_bytes = int(self.tensor.shape[1]) * self.tensor.element_size()
        if row_stride_bytes % 16 != 0:
            raise ValueError(
                f"kv-canary: RealKvSource.tensor dim-1 byte width must be a multiple of 16, "
                f"got {row_stride_bytes} bytes (shape={tuple(self.tensor.shape)}, "
                f"dtype={self.tensor.dtype})"
            )


@dataclass(frozen=True, slots=True, kw_only=True)
class VerifyOrWriteContext:

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a multiple of 16 within [16, num_bytes_per_token], e.g. min(64, num_bytes_per_token)
2. Derive read_bytes from num_bytes_per_token: read_bytes = num_bytes_per_token if num_bytes_per_token <= 64 else ((num_bytes_per_token // 4 + 15) // 16) * 16

Example fix

# before
src = RealKvSource(..., read_bytes=100)
# after
src = RealKvSource(..., read_bytes=96)

assert 0 < read_bytes <= num_bytes_per_token and read_bytes % 16 == 0, read_bytes

When it happens

Trigger: Setting read_bytes=0 (fingerprint nothing), read_bytes > num_bytes_per_token, or a non-multiple-of-16 sampling width when constructing the source.

Common situations: Tuning canary cost by sampling fewer bytes per token and choosing e.g. 100 instead of 96; copying read_bytes from a config expressed in elements not bytes.

Related errors

• kv-canary: RealKvSource.num_bytes_per_token must be a positi
• kv-canary: RealKvSource.tensor dim-1 byte width must be a mu
• v_cache must be provided
• kv-canary: {name} must be on {reference_name}'s device {refe
• kv-canary: launch_canary_plan_kernels_torch_reference verify

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0d51205eee1bdaa9. Report an issue: GitHub.