sgl-project/sglang · error · ValueError

HiSparse speculative swap requires 2-4 steps, got {num_steps

Error message

HiSparse speculative swap requires 2-4 steps, got {num_steps}.

What it means

The HiSparse speculative-decode cache swap kernel is compiled for a fixed speculative step count between 2 and 4; top_k_tokens.shape[1] (num_steps) outside that range has no kernel instantiation. The check reads the step dimension of the top-k tensor before launching.

Source

Thrown at python/sglang/kernels/ops/kvcache/hisparse.py:96

    device_buffer: torch.Tensor,
    top_k_device_locs: torch.Tensor,
    req_pool_indices: torch.Tensor,
    seq_lens: torch.Tensor,
    state: HiSparseSpecState,
    num_real_reqs: torch.Tensor,
    miss_src: torch.Tensor | None = None,
    miss_dst: torch.Tensor | None = None,
    miss_count: torch.Tensor | None = None,
) -> None:
    """Resolve all speculative steps and swap unique misses in one launch pair.

    Optional miss-plan outputs use the same protocol as the single-step HiSparse
    kernel, so shared-index layers can replay only the Host-to-GPU copies with
    ``copy_cache_planned_mla``.
    """
    _, num_steps, num_top_k = top_k_tokens.shape
    if not 2 <= num_steps <= 4:
        raise ValueError(
            f"HiSparse speculative swap requires 2-4 steps, got {num_steps}."
        )
    hot_buffer_size = state.cache_policy.size(1)
    page_size = device_buffer_tokens.size(1) - hot_buffer_size
    item_size_bytes = host_cache.stride(0) * host_cache.element_size()
    record_miss_plan = miss_src is not None
    if record_miss_plan:
        if miss_dst is None or miss_count is None:
            raise ValueError(
                "miss_src, miss_dst, and miss_count must be provided together."
            )
        if miss_src.dtype != torch.int64 or miss_dst.dtype != torch.int32:
            raise ValueError("miss_src must be int64 and miss_dst must be int32.")
        if miss_count.dtype != torch.int32:
            raise ValueError("miss_count must be int32.")
        plan_capacity = num_steps * num_top_k
        batch_size = top_k_tokens.size(0)
        if (

View on GitHub (pinned to 0132848349)

Solutions

1. Set the speculative step count to 2, 3, or 4
2. For num_steps == 1 use the single-step HiSparse kernel (non-spec API)
3. Verify top_k_tokens actually has [batch, steps, top_k] layout and you are not reading a different dim as steps

Example fix

# before (num_steps=1)
top_k_tokens = torch.zeros(bs, 1, topk, dtype=torch.int64)
load_cache_to_device_buffer_spec_mla(...)
# after (use single-step API)
load_cache_to_device_buffer_mla(...)  # non-spec variant

_, num_steps, _ = top_k_tokens.shape
assert 2 <= num_steps <= 4, num_steps

def spec_steps_supported(top_k_tokens: torch.Tensor) -> bool:
    return 2 <= top_k_tokens.shape[1] <= 4

try:
    load_cache_to_device_buffer_spec_mla(...)
except ValueError:
    load_cache_to_device_buffer_mla(...)  # single-step fallback

When it happens

Trigger: Calling load_cache_to_device_buffer_spec_mla with top_k_tokens of shape [batch, num_steps, top_k] where num_steps is 1 or >4 — e.g. speculative_draft_num_steps misconfigured outside 2..4.

Common situations: Setting --speculative-num-steps to 1 (use single-step API instead) or 5+; EAGLE/MTP configs with unsupported step counts; feeding a single-step top-k tensor into the spec API.

Related errors

• speculative miss_src/miss_dst must have shape [batch, >= ste
• speculative miss_count must have shape [batch].
• Invalid stacked fused KV projection shape: got {tuple(kv.sha
• Invalid fused KV projection shape: got {tuple(kv.shape)}, ex
• Invalid stacked k_norm_weight shape for fused KV materializa

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/62efcc2684393c8a. Report an issue: GitHub.