sgl-project/sglang · error · RuntimeError

RoPE cos/sin cache is too short for fused KV materialization

Error message

RoPE cos/sin cache is too short for fused KV materialization: max_position={max_position}, cache_len={int(cos_sin_cache.shape[0])}.

What it means

The RoPE cos_sin_cache (indexed by absolute position) must be longer than the maximum position being materialized. The code raises when max_position >= cos_sin_cache.shape[0].

Source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:353

        )

        if self.max_position_hint is not None:
            self._ensure_rope_cache(self.max_position_hint)

    def _ensure_rope_cache(self, max_position: int) -> torch.Tensor:
        if max_position + 1 > self._reserved_rope_cache_len:
            ensure_cos_sin_cache_length = getattr(
                self.rotary_emb, "_ensure_cos_sin_cache_length", None
            )
            if callable(ensure_cos_sin_cache_length):
                ensure_cos_sin_cache_length(max_position)
                self._reserved_rope_cache_len = int(
                    self.rotary_emb.cos_sin_cache.shape[0]
                )

        cos_sin_cache = self.rotary_emb.cos_sin_cache
        if max_position >= int(cos_sin_cache.shape[0]):
            raise RuntimeError(
                "RoPE cos/sin cache is too short for fused KV materialization: "
                f"max_position={max_position}, cache_len={int(cos_sin_cache.shape[0])}."
            )
        if cos_sin_cache.device != self.device:
            cos_sin_cache = cos_sin_cache.to(self.device)
        return cos_sin_cache

    def _ensure_workspace(self, total_ctx: int, dtype: torch.dtype) -> None:
        if (
            self._workspace_capacity >= total_ctx
            and self._workspace_dtype == dtype
            and self._proj_workspace is not None
            and self._k_workspace is not None
            and self._v_workspace is not None
        ):
            return

        new_capacity = max(1, total_ctx)

View on GitHub (pinned to 0132848349)

Solutions

1. Increase max_position_embeddings / rebuild the rotary_emb with a larger cos_sin_cache before materializing.
2. Truncate or chunk positions to stay below cache_len.
3. Check that _reserved_rope_cache_len reservations account for the full context length.

Example fix

// before
rotary_emb = RotaryEmbedding(head_dim, max_position=8192)  # positions reach 20000
// after
rotary_emb = RotaryEmbedding(head_dim, max_position=32768)

max_pos = int(positions.max())
assert max_pos < int(rotary_emb.cos_sin_cache.shape[0])

try:
    mat.materialize(...)
except RuntimeError as e:
    if 'cos/sin cache' in str(e):
        rotary_emb.extend_cache(...)  # or rebuild with larger max_position

When it happens

Trigger: Materializing KV for token positions at or beyond the cache length — e.g. long contexts exceeding the rotary_emb's max_position_embeddings-derived cache.

Common situations: Running contexts longer than the model's trained max_position_embeddings, or a dynamically extended context without rebuilding the rotary cache; also RoPE scaling changes that shrink the effective cache.

Related errors

• rope_pool_fused expects pool tensors to be 3-D
• v_pool shape must match k_pool shape, got {v_pool.shape} vs
• v_cache must be provided
• k_cache can only be None when only_qv=True
• rope_pool_fused expects q/k/v to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/32ba5273411e44b2. Report an issue: GitHub.