sgl-project/sglang · error · ValueError

speculative miss_count must have shape [batch].

Error message

speculative miss_count must have shape [batch].

What it means

miss_count is a per-request 1-D counter array of length >= batch size; the kernel increments one entry per request. A multi-dim tensor or a shorter-than-batch tensor cannot be indexed safely, so it is rejected. This is the last of the miss-plan shape checks before stride validation.

Source

Thrown at python/sglang/kernels/ops/kvcache/hisparse.py:127

            raise ValueError("miss_src must be int64 and miss_dst must be int32.")
        if miss_count.dtype != torch.int32:
            raise ValueError("miss_count must be int32.")
        plan_capacity = num_steps * num_top_k
        batch_size = top_k_tokens.size(0)
        if (
            miss_src.ndim != 2
            or miss_dst.ndim != 2
            or miss_src.size(0) < batch_size
            or miss_dst.size(0) < batch_size
            or miss_src.size(1) < plan_capacity
            or miss_dst.size(1) < plan_capacity
        ):
            raise ValueError(
                "speculative miss_src/miss_dst must have shape "
                f"[batch, >= steps * top_k] (capacity {plan_capacity})."
            )
        if miss_count.ndim != 1 or miss_count.numel() < batch_size:
            raise ValueError("speculative miss_count must have shape [batch].")
        if miss_src.stride(0) != miss_dst.stride(0):
            raise ValueError("miss_src/miss_dst row strides must match.")
    else:
        if miss_dst is not None or miss_count is not None:
            raise ValueError(
                "miss_src, miss_dst, and miss_count must be provided together."
            )
        empty = torch.empty(0)
        miss_src = miss_dst = miss_count = empty

    module = _jit_spec_module(
        item_size_bytes,
        _GATHER_BLOCK_SIZE,
        num_top_k,
        hot_buffer_size,
        num_steps,
        record_miss_plan,
    )

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate miss_count as a 1-D int32 tensor of length >= batch size
2. Reallocate or slice counters whenever the batch size changes
3. Keep counters and top_k_tokens batch allocations in one place so they resize together

Example fix

# before
miss_count = torch.zeros(bs, 1, dtype=torch.int32, device=dev)
# after
miss_count = torch.zeros(bs, dtype=torch.int32, device=dev)

batch = top_k_tokens.size(0)
assert miss_count.ndim == 1 and miss_count.numel() >= batch

def miss_count_shape_ok(top_k_tokens, miss_count) -> bool:
    return miss_count.ndim == 1 and miss_count.numel() >= top_k_tokens.size(0)

When it happens

Trigger: Calling load_cache_to_device_buffer_spec_mla with a miss-plan and miss_count.ndim != 1 or miss_count.numel() < top_k_tokens.size(0) — e.g. a scalar, a [batch,1] tensor, or a stale smaller batch counter.

Common situations: Batch size grows (dynamic batching / continuous batching) while miss_count was preallocated for the old batch; reshaping counters to 2-D for storage and forgetting to squeeze.

Related errors

• speculative miss_src/miss_dst must have shape [batch, >= ste
• HiSparse speculative swap requires 2-4 steps, got {num_steps
• miss_src must be int64 and miss_dst must be int32.
• miss_count must be int32.
• The pointers must be multiple of 16 bytes.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4dd606bd7480150e. Report an issue: GitHub.