sgl-project/sglang · error · ValueError

Cosmos3 batched prompts must tokenize to the same length…

Error message

Cosmos3 batched prompts must tokenize to the same length because GEN cross-attention does not mask padded text K/V; split prompts into equal-length batches instead (lengths={seq_lens})

What it means

The Cosmos3 stage tokenizes a batch of prompts and requires all tokenized sequence lengths to be identical after padding. Because the GEN cross-attention layers do not apply the text attention mask to padded K/V positions, unequal-length prompts would let pad tokens contaminate conditioning, so unequal batches are rejected outright.

Solutions

Split requests into batches where all prompts tokenize to the same length (bucket prompts by token count)
Process prompts one-at-a-time (batch size 1) if lengths are unpredictable
Pre-tokenize prompts client-side and group equal-length token sequences together

Example fix

# before
batches = [prompts]  # mixed lengths

# after
from collections import defaultdict
buckets = defaultdict(list)
for p in prompts:
    buckets[len(tokenizer(p).input_ids)].append(p)
batches = list(buckets.values())

Defensive patterns

Strategy: validation

Validate before calling

lens = [len(tok.input_ids) for tok in tokenizer(prompts)]
assert len(set(lens)) == 1, f'unequal token lengths: {lens}'

Type guard

def prompts_same_token_len(prompts: list[str], tok) -> bool:
    return len({len(tok(p).input_ids) for p in prompts}) == 1

Prevention

Bucket prompts by tokenized length before batching
Run with batch size 1 for heterogeneous prompts

When it happens

Trigger: Calling the Cosmos3 stage's forward() (which calls _tokenize_prompt) with a batch of text prompts whose tokenizer output lengths differ (after any padding computed in the loop).

Common situations: Batching video-generation requests with mixed prompt lengths, changing the tokenizer/truncation settings, or appending an empty/short prompt to a batch of long ones.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9d9c2e19ba005d87. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py:353

            # Reserve room for the two special tokens (EOS + vision_start) so the
            # final length cannot exceed ``max_sequence_length``.
            token_ids = token_ids[: max_sequence_length - 2]
            # Add EOS and vision_start tokens
            token_ids.append(self.tokenizer.eos_token_id)
            if vision_start_id is not None:
                token_ids.append(vision_start_id)

            seq_len = len(token_ids)
            pad_len = max_sequence_length - seq_len
            attention_mask = [1] * seq_len + [0] * pad_len
            token_ids = token_ids + [pad_token_id] * pad_len
            input_id_lists.append(token_ids)
            attention_mask_lists.append(attention_mask)
            seq_lens.append(seq_len)

        if len(set(seq_lens)) != 1:
            raise ValueError(
                "Cosmos3 batched prompts must tokenize to the same length because "
                "GEN cross-attention does not mask padded text K/V; split prompts "
                f"into equal-length batches instead (lengths={seq_lens})"
            )
        input_ids = torch.tensor(input_id_lists, dtype=torch.long, device=device)
        attention_mask = torch.tensor(
            attention_mask_lists, dtype=torch.long, device=device
        )
        return input_ids, attention_mask, seq_lens[0]

    def forward(self, batch: Req, server_args: ServerArgs) -> Req:
        """Tokenize prompt and negative prompt."""
        device = get_local_torch_device()
        prompt = batch.prompt
        negative_prompt = batch.negative_prompt or COSMOS3_DEFAULT_NEGATIVE_PROMPT

        # Get parameters
        max_sequence_length = getattr(batch, "max_sequence_length", None) or 512

View on GitHub (pinned to 0132848349)