sgl-project/sglang · error · ValueError

kv-canary: launch_canary_plan_kernels_torch_reference verify

Error message

kv-canary: launch_canary_plan_kernels_torch_reference verify_capacity={verify_capacity} does not match verify_plan_out.verify_slot_indices.shape[0]={plan_verify_capacity}

What it means

The Python reference implementation of launch_canary_plan_kernels requires that the verify_capacity argument exactly equals verify_plan_out.verify_slot_indices.shape[0]. The reference iterates over the plan output slots, so a mismatched capacity would silently compute wrong results; it raises instead. This mirrors a validation the real Triton path performs.

Source

Thrown at python/sglang/kernels/ops/kv_canary/plan_ref.py:33

    write_plan_out: WritePlan,
    req_pool_indices: torch.Tensor,
    prefix_lens: torch.Tensor,
    extend_seq_lens: torch.Tensor,
    req_to_token: torch.Tensor,
    swa_window_size: int,
    full_to_swa_index_mapping: Optional[torch.Tensor],
    verify_capacity: int,
    req_to_verify_expected_tokens: Optional[torch.Tensor],
    req_to_verify_expected_tokens_valid_lens: Optional[torch.Tensor],
    kv_token_id_vs_position_offset: int,
) -> None:
    """Python reference for :func:`launch_canary_plan_kernels`. Same signature & byte-equal semantics."""
    bs = int(req_pool_indices.shape[0])
    work_device = torch.device("cpu")

    plan_verify_capacity = int(verify_plan_out.verify_slot_indices.shape[0])
    if verify_capacity != plan_verify_capacity:
        raise ValueError(
            f"kv-canary: launch_canary_plan_kernels_torch_reference verify_capacity={verify_capacity} does not "
            f"match verify_plan_out.verify_slot_indices.shape[0]={plan_verify_capacity}"
        )
    write_req_capacity = int(write_plan_out.write_seed_slot_indices.shape[0])

    req_pool_indices_host = req_pool_indices.detach().to(
        device=work_device, dtype=torch.int64
    )
    prefix_lens_host = prefix_lens.detach().to(device=work_device, dtype=torch.int64)
    extend_seq_lens_host = extend_seq_lens.detach().to(
        device=work_device, dtype=torch.int64
    )
    req_to_token_host = req_to_token.detach().to(device=work_device, dtype=torch.int64)

    lut: Optional[torch.Tensor] = None
    if full_to_swa_index_mapping is not None:
        lut = full_to_swa_index_mapping.detach().to(device=work_device)

View on GitHub (pinned to 0132848349)

Solutions

Allocate/resize verify_plan_out.verify_slot_indices with shape[0] == verify_capacity before the call
Recompute verify_capacity from the same expression used when the plan output was created (e.g. bs * verify_topk)
If reusing output buffers, key them on (bs, verify_capacity) so stale sizes are never passed

Example fix

# before
verify_plan_out = alloc_verify_plan(bs * topk)
launch_ref(..., verify_capacity=bs * (topk + 1), ...)
# after
verify_capacity = bs * (topk + 1)
verify_plan_out = alloc_verify_plan(verify_capacity)
launch_ref(..., verify_capacity=verify_capacity, ...)

Defensive patterns

Strategy: validation

Validate before calling

assert verify_capacity == verify_plan_out.verify_slot_indices.shape[0], (
    verify_capacity, verify_plan_out.verify_slot_indices.shape[0])

Prevention

Derive verify_capacity and the output allocation from one shared expression
Do not reuse plan output buffers across config changes (topk/capacity)

When it happens

Trigger: Calling launch_canary_plan_kernels_torch_reference with verify_capacity taken from config/topk math while verify_plan_out was allocated with a different capacity (e.g. capacity=verify_topk but the output buffer sized to topk, or a stale reused output buffer).

Common situations: Reusing a cached verify_plan_out across configs after changing speculative-decoding topk/capacity; computing capacity as bs*topk while the output was allocated with only bs slots.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/715a7e956d1a0f52. Report an issue: GitHub.