{"record":{"id":"9d9c2e19ba005d87","repo":"sgl-project/sglang","slug":"cosmos3-batched-prompts-must-tokenize-to-the-same","errorCode":null,"errorMessage":"Cosmos3 batched prompts must tokenize to the same length because GEN cross-attention does not mask padded text K/V; split prompts into equal-length batches instead (lengths={seq_lens})","messagePattern":"Cosmos3 batched prompts must tokenize to the same length because GEN cross-attention does not mask padded text K/V; split prompts into equal-length batches instead \\(lengths=(.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py","lineNumber":353,"sourceCode":"\n            # Reserve room for the two special tokens (EOS + vision_start) so the\n            # final length cannot exceed ``max_sequence_length``.\n            token_ids = token_ids[: max_sequence_length - 2]\n            # Add EOS and vision_start tokens\n            token_ids.append(self.tokenizer.eos_token_id)\n            if vision_start_id is not None:\n                token_ids.append(vision_start_id)\n\n            seq_len = len(token_ids)\n            pad_len = max_sequence_length - seq_len\n            attention_mask = [1] * seq_len + [0] * pad_len\n            token_ids = token_ids + [pad_token_id] * pad_len\n            input_id_lists.append(token_ids)\n            attention_mask_lists.append(attention_mask)\n            seq_lens.append(seq_len)\n\n        if len(set(seq_lens)) != 1:\n            raise ValueError(\n                \"Cosmos3 batched prompts must tokenize to the same length because \"\n                \"GEN cross-attention does not mask padded text K/V; split prompts \"\n                f\"into equal-length batches instead (lengths={seq_lens})\"\n            )\n        input_ids = torch.tensor(input_id_lists, dtype=torch.long, device=device)\n        attention_mask = torch.tensor(\n            attention_mask_lists, dtype=torch.long, device=device\n        )\n        return input_ids, attention_mask, seq_lens[0]\n\n    def forward(self, batch: Req, server_args: ServerArgs) -> Req:\n        \"\"\"Tokenize prompt and negative prompt.\"\"\"\n        device = get_local_torch_device()\n        prompt = batch.prompt\n        negative_prompt = batch.negative_prompt or COSMOS3_DEFAULT_NEGATIVE_PROMPT\n\n        # Get parameters\n        max_sequence_length = getattr(batch, \"max_sequence_length\", None) or 512","sourceCodeStart":335,"sourceCodeEnd":371,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py#L335-L371","documentation":"The Cosmos3 stage tokenizes a batch of prompts and requires all tokenized sequence lengths to be identical after padding. Because the GEN cross-attention layers do not apply the text attention mask to padded K/V positions, unequal-length prompts would let pad tokens contaminate conditioning, so unequal batches are rejected outright.","triggerScenarios":"Calling the Cosmos3 stage's forward() (which calls _tokenize_prompt) with a batch of text prompts whose tokenizer output lengths differ (after any padding computed in the loop).","commonSituations":"Batching video-generation requests with mixed prompt lengths, changing the tokenizer/truncation settings, or appending an empty/short prompt to a batch of long ones.","solutions":["Split requests into batches where all prompts tokenize to the same length (bucket prompts by token count)","Process prompts one-at-a-time (batch size 1) if lengths are unpredictable","Pre-tokenize prompts client-side and group equal-length token sequences together"],"exampleFix":"# before\nbatches = [prompts]  # mixed lengths\n\n# after\nfrom collections import defaultdict\nbuckets = defaultdict(list)\nfor p in prompts:\n    buckets[len(tokenizer(p).input_ids)].append(p)\nbatches = list(buckets.values())","handlingStrategy":"validation","validationCode":"lens = [len(tok.input_ids) for tok in tokenizer(prompts)]\nassert len(set(lens)) == 1, f'unequal token lengths: {lens}'","typeGuard":"def prompts_same_token_len(prompts: list[str], tok) -> bool:\n    return len({len(tok(p).input_ids) for p in prompts}) == 1","tryCatchPattern":null,"preventionTips":["Bucket prompts by tokenized length before batching","Run with batch size 1 for heterogeneous prompts"],"tags":["cosmos3","multimodal-gen","batching","tokenization","validation"],"backgroundTag":"batch-length-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}