deepseek-ai/DeepSeek-V3 · error · AssertionError

Prompt length exceeds model maximum sequence length (max_seq

Error message

Prompt length exceeds model maximum sequence length (max_seq_len=${model.max_seq_len})

What it means

Thrown in generate() (inference/generate.py:52) before any decoding starts: the longest prompt (after chat template) must fit entirely within model.max_seq_len, since KV cache slots are preallocated for the full sequence. total_len is clamped to max_seq_len, so a prompt at/near the limit would leave no room for generation — but the assert only requires prompt_len <= max_seq_len. Note the boundary: a prompt exactly equal to max_seq_len passes the assert yet yields zero new tokens.

Source

Thrown at inference/generate.py:52

    max_new_tokens: int,
    eos_id: int,
    temperature: float = 1.0
) -> List[List[int]]:
    """
    Generates new tokens based on the given prompt tokens using the specified model.

    Args:
        model (Transformer): The transformer model used for token generation.
        prompt_tokens (List[List[int]]): A list of lists containing the prompt tokens for each sequence.
        max_new_tokens (int): The maximum number of new tokens to generate.
        eos_id (int): The end-of-sequence token ID.
        temperature (float, optional): The temperature value for sampling. Defaults to 1.0.

    Returns:
        List[List[int]]: A list of lists containing the generated tokens for each sequence.
    """
    prompt_lens = [len(t) for t in prompt_tokens]
    assert max(prompt_lens) <= model.max_seq_len, f"Prompt length exceeds model maximum sequence length (max_seq_len={model.max_seq_len})"
    total_len = min(model.max_seq_len, max_new_tokens + max(prompt_lens))
    tokens = torch.full((len(prompt_tokens), total_len), -1, dtype=torch.long, device="cuda")
    for i, t in enumerate(prompt_tokens):
        tokens[i, :len(t)] = torch.tensor(t, dtype=torch.long, device="cuda")
    prev_pos = 0
    finished = torch.tensor([False] * len(prompt_tokens), device="cuda")
    prompt_mask = tokens != -1
    for cur_pos in range(min(prompt_lens), total_len):
        logits = model.forward(tokens[:, prev_pos:cur_pos], prev_pos)
        if temperature > 0:
            next_token = sample(logits, temperature)
        else:
            next_token = logits.argmax(dim=-1)
        next_token = torch.where(prompt_mask[:, cur_pos], tokens[:, cur_pos], next_token)
        tokens[:, cur_pos] = next_token
        finished |= torch.logical_and(~prompt_mask[:, cur_pos], next_token == eos_id)
        prev_pos = cur_pos
        if finished.all():

View on GitHub (pinned to 9b4e9788e4)

Solutions

  1. Shorten the prompt / truncate earlier turns in the chat history before calling generate
  2. Increase max_seq_len in the model config JSON — bounded by the KV-cache memory your GPUs have
  3. For interactive mode, use the built-in /clear command to reset accumulated history
  4. Pre-check: if len(prompt_tokens[0]) > model.max_seq_len, trim or fail with your own message before calling generate

Example fix

# before
completion_tokens = generate(model, [prompt_tokens], max_new_tokens, eos_id, temperature)

# after
max_prompt = model.max_seq_len - max_new_tokens
if len(prompt_tokens) > max_prompt:
    prompt_tokens = prompt_tokens[:max_prompt]  # or raise a clear error
completion_tokens = generate(model, [prompt_tokens], max_new_tokens, eos_id, temperature)
Defensive patterns

Strategy: validation

Validate before calling

def safe_generate(model, prompt_tokens, max_new_tokens, eos_id, temperature=1.0):
    longest = max(len(t) for t in prompt_tokens)
    if longest + 1 > model.max_seq_len:  # leave room for >=1 new token
        raise ValueError(
            f"prompt len {longest} + 1 new token exceeds max_seq_len {model.max_seq_len}; "
            f"truncate the prompt or raise max_seq_len in the config"
        )
    return generate(model, prompt_tokens, max_new_tokens, eos_id, temperature)

Type guard

def prompt_fits(prompt_len: int, max_new_tokens: int, max_seq_len: int) -> bool:
    return prompt_len + max(1, max_new_tokens) <= max_seq_len

Try / catch

try:
    tokens = generate(model, prompts, max_new_tokens, eos_id, temp)
except AssertionError as e:
    if "max sequence length" in str(e):
        messages = messages[-2:]  # drop oldest turns and retry once
        prompts = [tokenizer.apply_chat_template(messages, add_generation_prompt=True)]
        tokens = generate(model, prompts, max_new_tokens, eos_id, temp)
    else:
        raise

Prevention

When it happens

Trigger: Calling generate(model, prompt_tokens, ...) where max(len(t) for t in prompt_tokens) > model.max_seq_len. Concretely: long pasted documents in --interactive chat with growing message history, or a long line in the --input-file batch, after the chat template adds its special tokens.

Common situations: Multi-turn interactive sessions where messages accumulate and each apply_chat_template call resends the whole history; feeding raw long documents without chunking; configs edited to a smaller max_seq_len than the data needs.

Related errors

AI-assisted analysis of deepseek-ai/DeepSeek-V3@9b4e9788e4 (2026-08-14). Data as JSON: /api/errors/24eccee753dd96e6. Report an issue: GitHub.