{"record":{"id":"7da82cf0c0998327","repo":"sgl-project/sglang","slug":"replicated-q-k-and-v-must-be-provided-together","errorCode":null,"errorMessage":"Replicated Q, K, and V must be provided together.","messagePattern":"Replicated Q, K, and V must be provided together\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/layer.py","lineNumber":402,"sourceCode":"\n    def _forward_with_kv_gather(\n        self,\n        q: torch.Tensor,\n        k: torch.Tensor,\n        v: torch.Tensor,\n        ctx_attn_metadata,\n        replicated_q: torch.Tensor | None,\n        replicated_k: torch.Tensor | None,\n        replicated_v: torch.Tensor | None,\n        seq_lens: list[int] | None,\n    ) -> tuple[torch.Tensor, torch.Tensor | None]:\n        if seq_lens is not None:\n            raise NotImplementedError(\n                \"K/V-gather SP does not support varlen UlyssesAttention.\"\n            )\n        if any(x is not None for x in (replicated_q, replicated_k, replicated_v)):\n            if any(x is None for x in (replicated_q, replicated_k, replicated_v)):\n                raise ValueError(\"Replicated Q, K, and V must be provided together.\")\n\n        k = sequence_model_parallel_all_gather(k, dim=1)\n        v = sequence_model_parallel_all_gather(v, dim=1)\n\n        local_query_len = q.shape[1]\n        if replicated_q is not None:\n            q = torch.cat([q, replicated_q], dim=1)\n            k = torch.cat([k, replicated_k], dim=1)\n            v = torch.cat([v, replicated_v], dim=1)\n\n        output = self.attn_impl.forward(q, k, v, ctx_attn_metadata)\n        if replicated_q is None:\n            return output, None\n        return output[:, :local_query_len], output[:, local_query_len:]\n\n    def forward(\n        self,\n        q: torch.Tensor,","sourceCodeStart":384,"sourceCodeEnd":420,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/layer.py#L384-L420","documentation":"In _forward_with_kv_gather, replicated Q, K, and V tensors (for replicated prefix/suffix tokens like text around an image) must be supplied together or not at all. A partial set would make the all-gather and output merge ill-defined, so the code validates the all-or-nothing invariant.","triggerScenarios":"Calling UlyssesAttention.forward with only some of replicated_q, replicated_k, replicated_v non-None (e.g. passing replicated_q for a text prefix but forgetting the corresponding K/V), with seq_lens None.","commonSituations":"Multimodal code paths that build replicated text tokens but skip K/V replication due to a conditional bug; refactoring that renamed one of the three arguments; a caller passing replicated_v=None intentionally for weight-only attention, which is not supported here.","solutions":["Pass all three of replicated_q, replicated_k, replicated_v, computed from the same replicated tokens","Or pass none of them if the batch has no replicated prefix/suffix segment","Fix the upstream caller that produces an incomplete triple (usually a conditional that builds q but not k/v)"],"exampleFix":"# before\nout, rep = attn(q, k, v, replicated_q=rep_q)\n# after\nout, rep = attn(q, k, v, replicated_q=rep_q, replicated_k=rep_k, replicated_v=rep_v)","handlingStrategy":"validation","validationCode":"if sum(x is not None for x in (replicated_q, replicated_k, replicated_v)) not in (0, 3):\n    raise ValueError(\"replicated_q/k/v must be all provided or all None\")","typeGuard":"def replicated_triple_ok(q, k, v) -> bool:\n    n = sum(x is not None for x in (q, k, v))\n    return n == 0 or n == 3","tryCatchPattern":null,"preventionTips":["Build replicated QKV in one place as a triple from the same tokens tuple","Add a unit test asserting the triple invariant for every multimodal batch builder"],"tags":["attention","replicated-tokens","argument-validation","multimodal"],"backgroundTag":"inconsistent-arguments","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}