{"record":{"id":"4dd606bd7480150e","repo":"sgl-project/sglang","slug":"speculative-miss-count-must-have-shape-batch","errorCode":null,"errorMessage":"speculative miss_count must have shape [batch].","messagePattern":"speculative miss_count must have shape \\[batch\\]\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/kvcache/hisparse.py","lineNumber":127,"sourceCode":"            raise ValueError(\"miss_src must be int64 and miss_dst must be int32.\")\n        if miss_count.dtype != torch.int32:\n            raise ValueError(\"miss_count must be int32.\")\n        plan_capacity = num_steps * num_top_k\n        batch_size = top_k_tokens.size(0)\n        if (\n            miss_src.ndim != 2\n            or miss_dst.ndim != 2\n            or miss_src.size(0) < batch_size\n            or miss_dst.size(0) < batch_size\n            or miss_src.size(1) < plan_capacity\n            or miss_dst.size(1) < plan_capacity\n        ):\n            raise ValueError(\n                \"speculative miss_src/miss_dst must have shape \"\n                f\"[batch, >= steps * top_k] (capacity {plan_capacity}).\"\n            )\n        if miss_count.ndim != 1 or miss_count.numel() < batch_size:\n            raise ValueError(\"speculative miss_count must have shape [batch].\")\n        if miss_src.stride(0) != miss_dst.stride(0):\n            raise ValueError(\"miss_src/miss_dst row strides must match.\")\n    else:\n        if miss_dst is not None or miss_count is not None:\n            raise ValueError(\n                \"miss_src, miss_dst, and miss_count must be provided together.\"\n            )\n        empty = torch.empty(0)\n        miss_src = miss_dst = miss_count = empty\n\n    module = _jit_spec_module(\n        item_size_bytes,\n        _GATHER_BLOCK_SIZE,\n        num_top_k,\n        hot_buffer_size,\n        num_steps,\n        record_miss_plan,\n    )","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/kvcache/hisparse.py#L109-L145","documentation":"miss_count is a per-request 1-D counter array of length >= batch size; the kernel increments one entry per request. A multi-dim tensor or a shorter-than-batch tensor cannot be indexed safely, so it is rejected. This is the last of the miss-plan shape checks before stride validation.","triggerScenarios":"Calling load_cache_to_device_buffer_spec_mla with a miss-plan and miss_count.ndim != 1 or miss_count.numel() < top_k_tokens.size(0) — e.g. a scalar, a [batch,1] tensor, or a stale smaller batch counter.","commonSituations":"Batch size grows (dynamic batching / continuous batching) while miss_count was preallocated for the old batch; reshaping counters to 2-D for storage and forgetting to squeeze.","solutions":["Allocate miss_count as a 1-D int32 tensor of length >= batch size","Reallocate or slice counters whenever the batch size changes","Keep counters and top_k_tokens batch allocations in one place so they resize together"],"exampleFix":"# before\nmiss_count = torch.zeros(bs, 1, dtype=torch.int32, device=dev)\n# after\nmiss_count = torch.zeros(bs, dtype=torch.int32, device=dev)","handlingStrategy":"validation","validationCode":"batch = top_k_tokens.size(0)\nassert miss_count.ndim == 1 and miss_count.numel() >= batch","typeGuard":"def miss_count_shape_ok(top_k_tokens, miss_count) -> bool:\n    return miss_count.ndim == 1 and miss_count.numel() >= top_k_tokens.size(0)","tryCatchPattern":null,"preventionTips":["Keep miss_count as a flat [batch] int32 tensor","Resize counters together with batch growth in continuous batching"],"tags":["hisparse","shape-validation","miss-plan"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}