sgl-project/sglang · error · ValueError

Cosmos3 batched prompts must tokenize to the same length bec

Error message

Cosmos3 batched prompts must tokenize to the same length because GEN cross-attention does not mask padded text K/V; split prompts into equal-length batches instead (lengths={seq_lens})

What it means

The Cosmos3 stage tokenizes a batch of prompts and requires all tokenized sequence lengths to be identical after padding. Because the GEN cross-attention layers do not apply the text attention mask to padded K/V positions, unequal-length prompts would let pad tokens contaminate conditioning, so unequal batches are rejected outright.

Source

Thrown at python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/cosmos3.py:353

            # Reserve room for the two special tokens (EOS + vision_start) so the
            # final length cannot exceed ``max_sequence_length``.
            token_ids = token_ids[: max_sequence_length - 2]
            # Add EOS and vision_start tokens
            token_ids.append(self.tokenizer.eos_token_id)
            if vision_start_id is not None:
                token_ids.append(vision_start_id)

            seq_len = len(token_ids)
            pad_len = max_sequence_length - seq_len
            attention_mask = [1] * seq_len + [0] * pad_len
            token_ids = token_ids + [pad_token_id] * pad_len
            input_id_lists.append(token_ids)
            attention_mask_lists.append(attention_mask)
            seq_lens.append(seq_len)

        if len(set(seq_lens)) != 1:
            raise ValueError(
                "Cosmos3 batched prompts must tokenize to the same length because "
                "GEN cross-attention does not mask padded text K/V; split prompts "
                f"into equal-length batches instead (lengths={seq_lens})"
            )
        input_ids = torch.tensor(input_id_lists, dtype=torch.long, device=device)
        attention_mask = torch.tensor(
            attention_mask_lists, dtype=torch.long, device=device
        )
        return input_ids, attention_mask, seq_lens[0]

    def forward(self, batch: Req, server_args: ServerArgs) -> Req:
        """Tokenize prompt and negative prompt."""
        device = get_local_torch_device()
        prompt = batch.prompt
        negative_prompt = batch.negative_prompt or COSMOS3_DEFAULT_NEGATIVE_PROMPT

        # Get parameters
        max_sequence_length = getattr(batch, "max_sequence_length", None) or 512

View on GitHub (pinned to 0132848349)

Solutions

1. Split requests into batches where all prompts tokenize to the same length (bucket prompts by token count)
2. Process prompts one-at-a-time (batch size 1) if lengths are unpredictable
3. Pre-tokenize prompts client-side and group equal-length token sequences together

Example fix

# before
batches = [prompts]  # mixed lengths

# after
from collections import defaultdict
buckets = defaultdict(list)
for p in prompts:
    buckets[len(tokenizer(p).input_ids)].append(p)
batches = list(buckets.values())

lens = [len(tok.input_ids) for tok in tokenizer(prompts)]
assert len(set(lens)) == 1, f'unequal token lengths: {lens}'

def prompts_same_token_len(prompts: list[str], tok) -> bool:
    return len({len(tok(p).input_ids) for p in prompts}) == 1

When it happens

Trigger: Calling the Cosmos3 stage's forward() (which calls _tokenize_prompt) with a batch of text prompts whose tokenizer output lengths differ (after any padding computed in the loop).

Common situations: Batching video-generation requests with mixed prompt lengths, changing the tokenizer/truncation settings, or appending an empty/short prompt to a batch of long ones.

Related errors

• Cosmos3 action input accepts one image field; use a list or
• Cosmos3 action prompt must be a string or non-empty list
• Cosmos3 batched action input requires one prompt per image,
• Cosmos3 action batch size {batch_size} exceeds --batching-ma
• batching config rule requires max_batch_size

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9d9c2e19ba005d87. Report an issue: GitHub.