sgl-project/sglang · error · ValueError

kv-canary: at most {consts.MAX_REAL_KV_SOURCES} RealKvSource

Error message

kv-canary: at most {consts.MAX_REAL_KV_SOURCES} RealKvSource entries supported by the CUDA ABI, got {len(real_kv_sources)}

What it means

The CUDA kernel ABI for canary verification fixes a static upper bound (consts.MAX_REAL_KV_SOURCES) on how many RealKvSource entries can be packed into the kernel parameter buffer. Passing more entries would overflow the ABI struct, so the Python launcher rejects the list up front.

Source

Thrown at python/sglang/kernels/ops/kv_canary/verify.py:309

        - Counters: each thread maintains a local count of active entries it processed, warp-reduces via
          ``__shfl_down_sync`` (offsets 16..1), then the warp leader (lane 0) does a single atomicAdd of the
          warp's summed count into slot_run_counter. kernel_run_counter += 1: single thread (tid == 0) does an
          atomicAdd once per launch.

    Calling contract:
        - Pure side-effect; never raises. Host polls violation_write_index[0] > 0 for is_errored and
          violation_ring[0] for the first violation.
        - kernel_run_counter is bumped every call (canary-ran health signal).
        - Safe in cuda-graph capture; caller refills plan in-place before replay.

    Pinned by torch reference
    :func:`sglang.kernels.ops.kv_canary.verify_ref.launch_canary_verify_kernel_torch_reference`; CUDA must match
    byte-for-byte.
    """
    canary_buf = context.canary_buf
    real_kv_sources = context.real_kv_sources
    if len(real_kv_sources) > consts.MAX_REAL_KV_SOURCES:
        raise ValueError(
            f"kv-canary: at most {consts.MAX_REAL_KV_SOURCES} RealKvSource entries supported by the CUDA ABI, "
            f"got {len(real_kv_sources)}"
        )

    _assert_contiguous(canary_buf, "canary_buf")
    _assert_contiguous(plan.verify_slot_indices, "plan.verify_slot_indices")
    _assert_contiguous(plan.verify_expected_tokens, "plan.verify_expected_tokens")
    _assert_contiguous(plan.verify_expected_positions, "plan.verify_expected_positions")
    _assert_contiguous(plan.verify_prev_slot_indices, "plan.verify_prev_slot_indices")
    _assert_contiguous(plan.verify_num_valid, "plan.verify_num_valid")
    _assert_contiguous(plan.enable, "plan.enable")
    _assert_contiguous(context.violation_ring, "violation_ring")
    _assert_contiguous(context.violation_write_index, "violation_write_index")
    _assert_contiguous(context.slot_run_counter, "slot_run_counter")
    _assert_contiguous(context.kernel_run_counter, "kernel_run_counter")

    padded_bufs, source_params = _build_real_kv_source_abi(
        real_kv_sources=real_kv_sources, device=canary_buf.device

View on GitHub (pinned to 0132848349)

Solutions

Reduce real_kv_sources to at most consts.MAX_REAL_KV_SOURCES entries (drop or merge the least important sources)
If more sources are genuinely needed, raise MAX_REAL_KV_SOURCES in consts and rebuild the matching CUDA kernel so the ABI agrees
Write a pre-flight check in your harness: assert len(ctx.real_kv_sources) <= consts.MAX_REAL_KV_SOURCES before launching

Example fix

# before
ctx = CanaryVerifyContext(real_kv_sources=[s1, s2, s3, s4, s5], ...)
launch_canary_verify_kernel(ctx, plan)
# after
assert len(sources) <= consts.MAX_REAL_KV_SOURCES
ctx = CanaryVerifyContext(real_kv_sources=sources[:consts.MAX_REAL_KV_SOURCES], ...)
launch_canary_verify_kernel(ctx, plan)

Defensive patterns

Strategy: validation

Validate before calling

from sglang.kernels.ops.kv_canary import consts
assert len(ctx.real_kv_sources) <= consts.MAX_REAL_KV_SOURCES
launch_canary_verify_kernel(ctx, plan)

Type guard

def sources_within_abi_limit(sources: list) -> bool:
    return len(sources) <= consts.MAX_REAL_KV_SOURCES

Try / catch

try:
    launch_canary_verify_kernel(ctx, plan)
except ValueError as e:
    if 'MAX_REAL_KV_SOURCES' in str(e):
        ctx.real_kv_sources = ctx.real_kv_sources[:consts.MAX_REAL_KV_SOURCES]
    else:
        raise

Prevention

Bound source lists at construction time
Re-run kernel build after changing consts.MAX_REAL_KV_SOURCES

When it happens

Trigger: Calling launch_canary_verify_kernel with a CanaryVerifyContext whose real_kv_sources list has more than consts.MAX_REAL_KV_SOURCES entries (typically >4).

Common situations: Wiring a canary harness to several KV pools / hybrid-cache sources (e.g. hierarchical cache layers + host cache) and exceeding the compiled limit; bumping the number of sources in a test sweep without recompiling the kernel with a larger ABI bound.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b7b44bc77e5365d3. Report an issue: GitHub.