sgl-project/sglang · error · ValueError

positions must match ctx_hidden token count for fused KV mat

Error message

positions must match ctx_hidden token count for fused KV materialization: positions={positions.numel()}, total_ctx={total_ctx}.

What it means

The positions tensor must contain exactly one entry per token in ctx_hidden (total_ctx). A length mismatch means RoPE would read garbage positions for some tokens.

Source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:403

        self._v_workspace = torch.empty_like(self._k_workspace)
        self._workspace_capacity = new_capacity
        self._workspace_dtype = dtype

    def materialize(
        self,
        ctx_hidden: torch.Tensor,
        positions: torch.Tensor,
        write_layer_kv: Callable[[int, torch.Tensor, torch.Tensor], None],
    ) -> None:
        """Materialize KV cache for all layers using batched projection."""
        total_ctx = ctx_hidden.shape[0]
        if total_ctx == 0:
            return

        if positions.ndim != 1:
            positions = positions.reshape(-1)
        if positions.numel() != total_ctx:
            raise ValueError(
                "positions must match ctx_hidden token count for fused KV materialization: "
                f"positions={positions.numel()}, total_ctx={total_ctx}."
            )

        if ctx_hidden.device != self.device:
            ctx_hidden = ctx_hidden.to(self.device, non_blocking=True)
        if ctx_hidden.dtype != self.flat_kv_weight_t.dtype:
            ctx_hidden = ctx_hidden.to(self.flat_kv_weight_t.dtype)
        if positions.device != self.device:
            positions = positions.to(
                device=self.device, dtype=torch.int64, non_blocking=True
            )
        elif positions.dtype != torch.int64:
            positions = positions.to(torch.int64)

        max_position = (
            self.max_position_hint
            if self.max_position_hint is not None

View on GitHub (pinned to 0132848349)

Solutions

1. Build positions as a flat 1D tensor of length ctx_hidden.shape[0] (or the token dim used for total_ctx).
2. Recompute positions after any truncation/extension of ctx_hidden.
3. Add an assert positions.numel() == ctx_hidden.shape[0] in your calling loop.

Example fix

// before
mat.materialize(ctx_hidden, positions[:-1])  # dropped last token
// after
mat.materialize(ctx_hidden, positions)  # len(positions) == ctx tokens

assert positions.reshape(-1).numel() == ctx_hidden.shape[0]

When it happens

Trigger: Calling materialize with positions sized for a different batch than the hidden states — e.g. positions from the draft model's step while ctx_hidden spans the full accepted context, or forgetting that positions gets flattened to 1D.

Common situations: Off-by-one after append/rollback in speculative decoding loops; passing [seq_len, batch] positions that flatten to a different count than tokens.

Related errors

• HiSparse speculative swap requires 2-4 steps, got {num_steps
• Invalid stacked fused KV projection shape: got {tuple(kv.sha
• Invalid fused KV projection shape: got {tuple(kv.shape)}, ex
• Invalid stacked k_norm_weight shape for fused KV materializa
• Invalid stacked eps shape for fused KV materialization: got

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ba34cd7e3a497550. Report an issue: GitHub.