sgl-project/sglang · error · ValueError

Replicated Q, K, and V must be provided together.

Error message

Replicated Q, K, and V must be provided together.

What it means

In _forward_with_kv_gather, replicated Q, K, and V tensors (for replicated prefix/suffix tokens like text around an image) must be supplied together or not at all. A partial set would make the all-gather and output merge ill-defined, so the code validates the all-or-nothing invariant.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/layer.py:402

    def _forward_with_kv_gather(
        self,
        q: torch.Tensor,
        k: torch.Tensor,
        v: torch.Tensor,
        ctx_attn_metadata,
        replicated_q: torch.Tensor | None,
        replicated_k: torch.Tensor | None,
        replicated_v: torch.Tensor | None,
        seq_lens: list[int] | None,
    ) -> tuple[torch.Tensor, torch.Tensor | None]:
        if seq_lens is not None:
            raise NotImplementedError(
                "K/V-gather SP does not support varlen UlyssesAttention."
            )
        if any(x is not None for x in (replicated_q, replicated_k, replicated_v)):
            if any(x is None for x in (replicated_q, replicated_k, replicated_v)):
                raise ValueError("Replicated Q, K, and V must be provided together.")

        k = sequence_model_parallel_all_gather(k, dim=1)
        v = sequence_model_parallel_all_gather(v, dim=1)

        local_query_len = q.shape[1]
        if replicated_q is not None:
            q = torch.cat([q, replicated_q], dim=1)
            k = torch.cat([k, replicated_k], dim=1)
            v = torch.cat([v, replicated_v], dim=1)

        output = self.attn_impl.forward(q, k, v, ctx_attn_metadata)
        if replicated_q is None:
            return output, None
        return output[:, :local_query_len], output[:, local_query_len:]

    def forward(
        self,
        q: torch.Tensor,

View on GitHub (pinned to 0132848349)

Solutions

1. Pass all three of replicated_q, replicated_k, replicated_v, computed from the same replicated tokens
2. Or pass none of them if the batch has no replicated prefix/suffix segment
3. Fix the upstream caller that produces an incomplete triple (usually a conditional that builds q but not k/v)

Example fix

# before
out, rep = attn(q, k, v, replicated_q=rep_q)
# after
out, rep = attn(q, k, v, replicated_q=rep_q, replicated_k=rep_k, replicated_v=rep_v)

if sum(x is not None for x in (replicated_q, replicated_k, replicated_v)) not in (0, 3):
    raise ValueError("replicated_q/k/v must be all provided or all None")

def replicated_triple_ok(q, k, v) -> bool:
    n = sum(x is not None for x in (q, k, v))
    return n == 0 or n == 3

When it happens

Trigger: Calling UlyssesAttention.forward with only some of replicated_q, replicated_k, replicated_v non-None (e.g. passing replicated_q for a text prefix but forgetting the corresponding K/V), with seq_lens None.

Common situations: Multimodal code paths that build replicated text tokens but skip K/V replication due to a conditional bug; refactoring that renamed one of the three arguments; a caller passing replicated_v=None intentionally for weight-only attention, which is not supported here.

Related errors

• Mode must be one of {valid_modes}, got {mode}
• mask_search_files_path_pos, mask_search_files_path_neg, and
• All ranges must be within [0, {max_seqlen}], got {range_valu
• USPAttention's masked path does not support replicated prefi
• LTX2DurationHead requires at least one of video_tokens / aud

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7da82cf0c0998327. Report an issue: GitHub.