{"record":{"id":"ba34cd7e3a497550","repo":"sgl-project/sglang","slug":"positions-must-match-ctx-hidden-token-count-for-fu","errorCode":null,"errorMessage":"positions must match ctx_hidden token count for fused KV materialization: positions={positions.numel()}, total_ctx={total_ctx}.","messagePattern":"positions must match ctx_hidden token count for fused KV materialization: positions=(.+?), total_ctx=(.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/speculative/fused_kv_materialize.py","lineNumber":403,"sourceCode":"        self._v_workspace = torch.empty_like(self._k_workspace)\n        self._workspace_capacity = new_capacity\n        self._workspace_dtype = dtype\n\n    def materialize(\n        self,\n        ctx_hidden: torch.Tensor,\n        positions: torch.Tensor,\n        write_layer_kv: Callable[[int, torch.Tensor, torch.Tensor], None],\n    ) -> None:\n        \"\"\"Materialize KV cache for all layers using batched projection.\"\"\"\n        total_ctx = ctx_hidden.shape[0]\n        if total_ctx == 0:\n            return\n\n        if positions.ndim != 1:\n            positions = positions.reshape(-1)\n        if positions.numel() != total_ctx:\n            raise ValueError(\n                \"positions must match ctx_hidden token count for fused KV materialization: \"\n                f\"positions={positions.numel()}, total_ctx={total_ctx}.\"\n            )\n\n        if ctx_hidden.device != self.device:\n            ctx_hidden = ctx_hidden.to(self.device, non_blocking=True)\n        if ctx_hidden.dtype != self.flat_kv_weight_t.dtype:\n            ctx_hidden = ctx_hidden.to(self.flat_kv_weight_t.dtype)\n        if positions.device != self.device:\n            positions = positions.to(\n                device=self.device, dtype=torch.int64, non_blocking=True\n            )\n        elif positions.dtype != torch.int64:\n            positions = positions.to(torch.int64)\n\n        max_position = (\n            self.max_position_hint\n            if self.max_position_hint is not None","sourceCodeStart":385,"sourceCodeEnd":421,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/speculative/fused_kv_materialize.py#L385-L421","documentation":"The positions tensor must contain exactly one entry per token in ctx_hidden (total_ctx). A length mismatch means RoPE would read garbage positions for some tokens.","triggerScenarios":"Calling materialize with positions sized for a different batch than the hidden states — e.g. positions from the draft model's step while ctx_hidden spans the full accepted context, or forgetting that positions gets flattened to 1D.","commonSituations":"Off-by-one after append/rollback in speculative decoding loops; passing [seq_len, batch] positions that flatten to a different count than tokens.","solutions":["Build positions as a flat 1D tensor of length ctx_hidden.shape[0] (or the token dim used for total_ctx).","Recompute positions after any truncation/extension of ctx_hidden.","Add an assert positions.numel() == ctx_hidden.shape[0] in your calling loop."],"exampleFix":"// before\nmat.materialize(ctx_hidden, positions[:-1])  # dropped last token\n// after\nmat.materialize(ctx_hidden, positions)  # len(positions) == ctx tokens","handlingStrategy":"validation","validationCode":"assert positions.reshape(-1).numel() == ctx_hidden.shape[0]","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Regenerate positions from the token metadata at the same place ctx_hidden is built.","Add asserts in speculative loop after append/rollback."],"tags":["shape-validation","speculative-decoding"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}