sgl-project/sglang · error · ValueError

speculative miss_src/miss_dst must have shape [batch, >= ste

Error message

speculative miss_src/miss_dst must have shape [batch, >= steps * top_k] (capacity {plan_capacity}).

What it means

The miss-plan src/dst buffers must be 2-D with at least batch rows and at least steps*top_k columns, because in the worst case every speculative token of every step misses. The check enforces ndim==2, size(0)>=batch and size(1)>=num_steps*num_top_k (the reported plan_capacity).

Source

Thrown at python/sglang/kernels/ops/kvcache/hisparse.py:122

        if miss_dst is None or miss_count is None:
            raise ValueError(
                "miss_src, miss_dst, and miss_count must be provided together."
            )
        if miss_src.dtype != torch.int64 or miss_dst.dtype != torch.int32:
            raise ValueError("miss_src must be int64 and miss_dst must be int32.")
        if miss_count.dtype != torch.int32:
            raise ValueError("miss_count must be int32.")
        plan_capacity = num_steps * num_top_k
        batch_size = top_k_tokens.size(0)
        if (
            miss_src.ndim != 2
            or miss_dst.ndim != 2
            or miss_src.size(0) < batch_size
            or miss_dst.size(0) < batch_size
            or miss_src.size(1) < plan_capacity
            or miss_dst.size(1) < plan_capacity
        ):
            raise ValueError(
                "speculative miss_src/miss_dst must have shape "
                f"[batch, >= steps * top_k] (capacity {plan_capacity})."
            )
        if miss_count.ndim != 1 or miss_count.numel() < batch_size:
            raise ValueError("speculative miss_count must have shape [batch].")
        if miss_src.stride(0) != miss_dst.stride(0):
            raise ValueError("miss_src/miss_dst row strides must match.")
    else:
        if miss_dst is not None or miss_count is not None:
            raise ValueError(
                "miss_src, miss_dst, and miss_count must be provided together."
            )
        empty = torch.empty(0)
        miss_src = miss_dst = miss_count = empty

    module = _jit_spec_module(
        item_size_bytes,
        _GATHER_BLOCK_SIZE,

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate miss_src/miss_dst as [batch, num_steps * top_k] (or larger), int64/int32 respectively
2. Grow plan buffers whenever batch size or spec steps/top_k change
3. Assert shapes before launch in your harness

Example fix

# before
miss_src = torch.zeros(bs, topk, dtype=torch.int64, device=dev)
# after
cap = num_steps * topk
miss_src = torch.zeros(bs, cap, dtype=torch.int64, device=dev)

batch = top_k_tokens.size(0); cap = num_steps * top_k_tokens.size(2)
assert miss_src.ndim == 2 and miss_src.size(0) >= batch and miss_src.size(1) >= cap
assert miss_dst.ndim == 2 and miss_dst.size(0) >= batch and miss_dst.size(1) >= cap

def miss_plan_capacity_ok(top_k_tokens, miss_src, miss_dst) -> bool:
    cap = top_k_tokens.shape[1] * top_k_tokens.shape[2]
    b = top_k_tokens.size(0)
    return (miss_src.ndim == 2 and miss_src.size(0) >= b and miss_src.size(1) >= cap
            and miss_dst.ndim == 2 and miss_dst.size(0) >= b and miss_dst.size(1) >= cap)

When it happens

Trigger: Calling load_cache_to_device_buffer_spec_mla with miss_src/miss_dst that are 1-D, have fewer rows than top_k_tokens.size(0), or fewer columns than num_steps*num_top_k (e.g. allocated for the single-step capacity only).

Common situations: Reusing single-step miss buffers in the spec path; allocating capacity = top_k instead of steps*top_k; batch-size growth after warmup with preallocated plan tensors.

Related errors

• speculative miss_count must have shape [batch].
• HiSparse speculative swap requires 2-4 steps, got {num_steps
• miss_src must be int64 and miss_dst must be int32.
• miss_count must be int32.
• The pointers must be multiple of 16 bytes.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8ad0653a0e78188b. Report an issue: GitHub.