sgl-project/sglang · error · ValueError

miss_count must be int32.

Error message

miss_count must be int32.

What it means

The third dtype check in the miss-plan triple: miss_count must be int32 because the kernel writes per-batch miss counts as 32-bit values. int64 or int16 counters are rejected to avoid ABI mismatch on kernel write-back.

Source

Thrown at python/sglang/kernels/ops/kvcache/hisparse.py:111

    """
    _, num_steps, num_top_k = top_k_tokens.shape
    if not 2 <= num_steps <= 4:
        raise ValueError(
            f"HiSparse speculative swap requires 2-4 steps, got {num_steps}."
        )
    hot_buffer_size = state.cache_policy.size(1)
    page_size = device_buffer_tokens.size(1) - hot_buffer_size
    item_size_bytes = host_cache.stride(0) * host_cache.element_size()
    record_miss_plan = miss_src is not None
    if record_miss_plan:
        if miss_dst is None or miss_count is None:
            raise ValueError(
                "miss_src, miss_dst, and miss_count must be provided together."
            )
        if miss_src.dtype != torch.int64 or miss_dst.dtype != torch.int32:
            raise ValueError("miss_src must be int64 and miss_dst must be int32.")
        if miss_count.dtype != torch.int32:
            raise ValueError("miss_count must be int32.")
        plan_capacity = num_steps * num_top_k
        batch_size = top_k_tokens.size(0)
        if (
            miss_src.ndim != 2
            or miss_dst.ndim != 2
            or miss_src.size(0) < batch_size
            or miss_dst.size(0) < batch_size
            or miss_src.size(1) < plan_capacity
            or miss_dst.size(1) < plan_capacity
        ):
            raise ValueError(
                "speculative miss_src/miss_dst must have shape "
                f"[batch, >= steps * top_k] (capacity {plan_capacity})."
            )
        if miss_count.ndim != 1 or miss_count.numel() < batch_size:
            raise ValueError("speculative miss_count must have shape [batch].")
        if miss_src.stride(0) != miss_dst.stride(0):
            raise ValueError("miss_src/miss_dst row strides must match.")

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate miss_count with dtype=torch.int32
2. If you track counts in int64 elsewhere, add a separate int32 staging tensor for the kernel

Example fix

# before
miss_count = torch.zeros(bs, dtype=torch.int64, device=dev)
# after
miss_count = torch.zeros(bs, dtype=torch.int32, device=dev)

assert miss_count.dtype == torch.int32

def miss_count_dtype_ok(c) -> bool:
    return c is None or c.dtype == torch.int32

When it happens

Trigger: Calling load_cache_to_device_buffer_spec_mla with a miss-plan and miss_count.dtype != torch.int32 (commonly torch.int64).

Common situations: Reusing an int64 counter tensor from elsewhere in the pipeline; allocating all plan tensors as int64 for uniformity.

Related errors

• miss_src must be int64 and miss_dst must be int32.
• speculative miss_src/miss_dst must have shape [batch, >= ste
• speculative miss_count must have shape [batch].
• {name} must have dtype {dtype}, got {t.dtype}
• kv-canary: scatter_req_token_ids flat_in must be int64, got

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a8dea6ae160414ee. Report an issue: GitHub.