sgl-project/sglang · error · ValueError

kv-canary: read_bytes must be <= num_bytes_per_token ({num_b

Error message

kv-canary: read_bytes must be <= num_bytes_per_token ({num_bytes_per_token}), got {requested}

What it means

kv-canary rejects a read_bytes larger than the per-token KV cell size. The canary only reads within one token's slot of the real KV pool, so requesting more than num_bytes_per_token is invalid and raises ValueError from _clip_read_bytes_aligned.

Source

Thrown at python/sglang/srt/kv_canary/pool_patcher/buffer_alloc.py:53

def _clip_read_bytes_aligned(*, requested: int, num_bytes_per_token: int) -> int:
    """Validate and clip read_bytes for the CUDA fold kernel's 128-bit aligned loads.

    Normalizes sentinels (``sys.maxsize`` -> ``num_bytes_per_token``, ``0`` -> ``0``) and
    rejects negative / unaligned / oversized requests.
    """
    if num_bytes_per_token <= 0 or num_bytes_per_token % _REAL_KV_READ_ALIGN != 0:
        raise ValueError(
            "kv-canary: num_bytes_per_token must be a positive multiple of "
            f"{_REAL_KV_READ_ALIGN}, got {num_bytes_per_token}"
        )
    if requested == 0:
        return 0
    if requested == sys.maxsize:
        return num_bytes_per_token
    if requested < 0:
        raise ValueError(f"kv-canary: read_bytes must be non-negative, got {requested}")
    if requested > num_bytes_per_token:
        raise ValueError(
            "kv-canary: read_bytes must be <= num_bytes_per_token "
            f"({num_bytes_per_token}), got {requested}"
        )
    if requested % _REAL_KV_READ_ALIGN != 0:
        raise ValueError(
            "kv-canary: read_bytes must be a multiple of "
            f"{_REAL_KV_READ_ALIGN}, got {requested}"
        )
    return requested


def make_row_source(
    *,
    layer_buffer: torch.Tensor,
    read_bytes: int,
) -> Tuple[RealKvSource, ...]:
    contiguous = layer_buffer.contiguous()
    num_slots = int(contiguous.shape[0])

View on GitHub (pinned to 0132848349)

Solutions

  1. Pass sys.maxsize to request the full per-token size instead of a hardcoded number
  2. Derive read_bytes from num_bytes_per_token (e.g. // 2 or an aligned fraction) rather than a constant
  3. Verify the pool's cell size (kv_head_num * head_size * dtype_bytes) before choosing read_bytes

Example fix

// before
src = make_row_source(pool, read_bytes=128)
// after
src = make_row_source(pool, read_bytes=sys.maxsize)  # full per-token size, auto-clipped
Defensive patterns

Strategy: validation

Validate before calling

read_bytes = min(read_bytes, pool.num_bytes_per_token)
# or simply request full size:
read_bytes = sys.maxsize

Type guard

def fits_cell(n: int, cell: int) -> bool:
    return 0 <= n <= cell

Prevention

When it happens

Trigger: Calling make_row_source or make_packed_source with read_bytes greater than the pool's num_bytes_per_token, e.g. hardcoded 128 while the layer's per-token cell is 64 bytes.

Common situations: Hardcoding a read size tuned for one layer/dtype and reusing it against a pool with a smaller cell (fp8 KV cache, fewer KV heads after TP); unit mismatch (bits vs bytes).

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/67ef17f1622d3079. Report an issue: GitHub.