{"record":{"id":"67ef17f1622d3079","repo":"sgl-project/sglang","slug":"kv-canary-read-bytes-must-be-num-bytes-per-tok","errorCode":null,"errorMessage":"kv-canary: read_bytes must be <= num_bytes_per_token ({num_bytes_per_token}), got {requested}","messagePattern":"kv-canary: read_bytes must be <= num_bytes_per_token \\((.+?)\\), got (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/kv_canary/pool_patcher/buffer_alloc.py","lineNumber":53,"sourceCode":"def _clip_read_bytes_aligned(*, requested: int, num_bytes_per_token: int) -> int:\n    \"\"\"Validate and clip read_bytes for the CUDA fold kernel's 128-bit aligned loads.\n\n    Normalizes sentinels (``sys.maxsize`` -> ``num_bytes_per_token``, ``0`` -> ``0``) and\n    rejects negative / unaligned / oversized requests.\n    \"\"\"\n    if num_bytes_per_token <= 0 or num_bytes_per_token % _REAL_KV_READ_ALIGN != 0:\n        raise ValueError(\n            \"kv-canary: num_bytes_per_token must be a positive multiple of \"\n            f\"{_REAL_KV_READ_ALIGN}, got {num_bytes_per_token}\"\n        )\n    if requested == 0:\n        return 0\n    if requested == sys.maxsize:\n        return num_bytes_per_token\n    if requested < 0:\n        raise ValueError(f\"kv-canary: read_bytes must be non-negative, got {requested}\")\n    if requested > num_bytes_per_token:\n        raise ValueError(\n            \"kv-canary: read_bytes must be <= num_bytes_per_token \"\n            f\"({num_bytes_per_token}), got {requested}\"\n        )\n    if requested % _REAL_KV_READ_ALIGN != 0:\n        raise ValueError(\n            \"kv-canary: read_bytes must be a multiple of \"\n            f\"{_REAL_KV_READ_ALIGN}, got {requested}\"\n        )\n    return requested\n\n\ndef make_row_source(\n    *,\n    layer_buffer: torch.Tensor,\n    read_bytes: int,\n) -> Tuple[RealKvSource, ...]:\n    contiguous = layer_buffer.contiguous()\n    num_slots = int(contiguous.shape[0])","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/kv_canary/pool_patcher/buffer_alloc.py#L35-L71","documentation":"kv-canary rejects a read_bytes larger than the per-token KV cell size. The canary only reads within one token's slot of the real KV pool, so requesting more than num_bytes_per_token is invalid and raises ValueError from _clip_read_bytes_aligned.","triggerScenarios":"Calling make_row_source or make_packed_source with read_bytes greater than the pool's num_bytes_per_token, e.g. hardcoded 128 while the layer's per-token cell is 64 bytes.","commonSituations":"Hardcoding a read size tuned for one layer/dtype and reusing it against a pool with a smaller cell (fp8 KV cache, fewer KV heads after TP); unit mismatch (bits vs bytes).","solutions":["Pass sys.maxsize to request the full per-token size instead of a hardcoded number","Derive read_bytes from num_bytes_per_token (e.g. // 2 or an aligned fraction) rather than a constant","Verify the pool's cell size (kv_head_num * head_size * dtype_bytes) before choosing read_bytes"],"exampleFix":"// before\nsrc = make_row_source(pool, read_bytes=128)\n// after\nsrc = make_row_source(pool, read_bytes=sys.maxsize)  # full per-token size, auto-clipped","handlingStrategy":"validation","validationCode":"read_bytes = min(read_bytes, pool.num_bytes_per_token)\n# or simply request full size:\nread_bytes = sys.maxsize","typeGuard":"def fits_cell(n: int, cell: int) -> bool:\n    return 0 <= n <= cell","tryCatchPattern":null,"preventionTips":["Derive read_bytes from num_bytes_per_token, never hardcode","Use sys.maxsize for the full-cell default"],"tags":["kv-canary","validation","valueerror","read-bytes"],"backgroundTag":"argument-validation-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}