sgl-project/sglang · error · ValueError

kv-canary: at most {consts.MAX_REAL_KV_SOURCES} RealKvSource

Error message

kv-canary: at most {consts.MAX_REAL_KV_SOURCES} RealKvSource entries supported by the CUDA ABI, got {len(real_kv_sources)}

What it means

The canary write kernel shares the same fixed CUDA ABI parameter table as the verify kernel, which only has room for consts.MAX_REAL_KV_SOURCES RealKvSource descriptors. The Python launcher refuses a longer list before touching the kernel.

Source

Thrown at python/sglang/kernels/ops/kv_canary/write.py:189

        - record_violation() identical to verify (atomicAdd + atomic-write).
        - Counters: thread of block 0 does atomicAdd(kernel_run_counter, 1); each block accumulates its
          entry_count and atomicAdds to slot_run_counter once at exit.

    Calling contract:
        - Pure side-effect; never raises.
        - Input-verification mismatch records violations but does NOT abort the chain.
        - kernel_run_counter is bumped every call.
        - Safe in cuda-graph capture; caller refills input_ids / positions / out_cache_loc / plan
          in-place before replay.

    Pinned by torch reference
    :func:`sglang.kernels.ops.kv_canary.write_ref.launch_canary_write_kernel_torch_reference`; CUDA must match
    byte-for-byte.
    """
    canary_buf = context.canary_buf
    real_kv_sources = context.real_kv_sources
    if len(real_kv_sources) > consts.MAX_REAL_KV_SOURCES:
        raise ValueError(
            f"kv-canary: at most {consts.MAX_REAL_KV_SOURCES} RealKvSource entries supported by the CUDA ABI, "
            f"got {len(real_kv_sources)}"
        )

    _assert_contiguous(canary_buf, "canary_buf")
    _assert_contiguous(plan.write_offsets, "plan.write_offsets")
    _assert_contiguous(plan.write_seed_slot_indices, "plan.write_seed_slot_indices")
    _assert_contiguous(plan.write_num_valid_reqs, "plan.write_num_valid_reqs")
    _assert_contiguous(input_ids, "input_ids")
    _assert_contiguous(positions, "positions")
    _assert_contiguous(out_cache_loc, "out_cache_loc")
    if enable_write_input_assert:
        if expected_input_tokens is None or expected_input_positions is None:
            raise ValueError(
                "kv-canary: expected input tensors are required when enable_write_input_assert=True"
            )
        _assert_contiguous(expected_input_tokens, "expected_input_tokens")
        _assert_contiguous(expected_input_positions, "expected_input_positions")

View on GitHub (pinned to 0132848349)

Solutions

Trim/merge real_kv_sources to <= consts.MAX_REAL_KV_SOURCES
If the extra sources are required, raise consts.MAX_REAL_KV_SOURCES and rebuild the CUDA extension so the ABI matches
Pre-assert the bound in your harness setup

Example fix

# before
ctx = CanaryWriteContext(real_kv_sources=all_sources)  # 5 entries
# after
ctx = CanaryWriteContext(real_kv_sources=all_sources[:consts.MAX_REAL_KV_SOURCES])

Defensive patterns

Strategy: validation

Validate before calling

from sglang.kernels.ops.kv_canary import consts
if len(ctx.real_kv_sources) > consts.MAX_REAL_KV_SOURCES:
    ctx.real_kv_sources = ctx.real_kv_sources[:consts.MAX_REAL_KV_SOURCES]
launch_canary_write_kernel(ctx, plan)

Type guard

def sources_within_abi_limit(sources: list) -> bool:
    return len(sources) <= consts.MAX_REAL_KV_SOURCES

Try / catch

try:
    launch_canary_write_kernel(ctx, plan)
except ValueError as e:
    if 'MAX_REAL_KV_SOURCES' not in str(e):
        raise
    ctx.real_kv_sources = ctx.real_kv_sources[:consts.MAX_REAL_KV_SOURCES]
    launch_canary_write_kernel(ctx, plan)

Prevention

Enforce the ABI bound where the context is built
Keep verify and write contexts sharing one bounded source list

When it happens

Trigger: Calling launch_canary_write_kernel with context.real_kv_sources longer than consts.MAX_REAL_KV_SOURCES (commonly >4).

Common situations: Attaching the canary write path to many KV sources (multiple cache tiers, DP shards folded into one context) and exceeding the compiled bound; tests enumerating sources in a sweep.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8f6de0282921311a. Report an issue: GitHub.