{"record":{"id":"24eccee753dd96e6","repo":"deepseek-ai/DeepSeek-V3","slug":"prompt-length-exceeds-model-maximum-sequence-lengt","errorCode":null,"errorMessage":"Prompt length exceeds model maximum sequence length (max_seq_len=${model.max_seq_len})","messagePattern":"Prompt length exceeds model maximum sequence length \\(max_seq_len=(.+?)\\)","errorType":"validation","errorClass":"AssertionError","httpStatus":null,"severity":"error","filePath":"inference/generate.py","lineNumber":52,"sourceCode":"    max_new_tokens: int,\n    eos_id: int,\n    temperature: float = 1.0\n) -> List[List[int]]:\n    \"\"\"\n    Generates new tokens based on the given prompt tokens using the specified model.\n\n    Args:\n        model (Transformer): The transformer model used for token generation.\n        prompt_tokens (List[List[int]]): A list of lists containing the prompt tokens for each sequence.\n        max_new_tokens (int): The maximum number of new tokens to generate.\n        eos_id (int): The end-of-sequence token ID.\n        temperature (float, optional): The temperature value for sampling. Defaults to 1.0.\n\n    Returns:\n        List[List[int]]: A list of lists containing the generated tokens for each sequence.\n    \"\"\"\n    prompt_lens = [len(t) for t in prompt_tokens]\n    assert max(prompt_lens) <= model.max_seq_len, f\"Prompt length exceeds model maximum sequence length (max_seq_len={model.max_seq_len})\"\n    total_len = min(model.max_seq_len, max_new_tokens + max(prompt_lens))\n    tokens = torch.full((len(prompt_tokens), total_len), -1, dtype=torch.long, device=\"cuda\")\n    for i, t in enumerate(prompt_tokens):\n        tokens[i, :len(t)] = torch.tensor(t, dtype=torch.long, device=\"cuda\")\n    prev_pos = 0\n    finished = torch.tensor([False] * len(prompt_tokens), device=\"cuda\")\n    prompt_mask = tokens != -1\n    for cur_pos in range(min(prompt_lens), total_len):\n        logits = model.forward(tokens[:, prev_pos:cur_pos], prev_pos)\n        if temperature > 0:\n            next_token = sample(logits, temperature)\n        else:\n            next_token = logits.argmax(dim=-1)\n        next_token = torch.where(prompt_mask[:, cur_pos], tokens[:, cur_pos], next_token)\n        tokens[:, cur_pos] = next_token\n        finished |= torch.logical_and(~prompt_mask[:, cur_pos], next_token == eos_id)\n        prev_pos = cur_pos\n        if finished.all():","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/generate.py#L34-L70","documentation":"Thrown in generate() (inference/generate.py:52) before any decoding starts: the longest prompt (after chat template) must fit entirely within model.max_seq_len, since KV cache slots are preallocated for the full sequence. total_len is clamped to max_seq_len, so a prompt at/near the limit would leave no room for generation — but the assert only requires prompt_len <= max_seq_len. Note the boundary: a prompt exactly equal to max_seq_len passes the assert yet yields zero new tokens.","triggerScenarios":"Calling generate(model, prompt_tokens, ...) where max(len(t) for t in prompt_tokens) > model.max_seq_len. Concretely: long pasted documents in --interactive chat with growing message history, or a long line in the --input-file batch, after the chat template adds its special tokens.","commonSituations":"Multi-turn interactive sessions where messages accumulate and each apply_chat_template call resends the whole history; feeding raw long documents without chunking; configs edited to a smaller max_seq_len than the data needs.","solutions":["Shorten the prompt / truncate earlier turns in the chat history before calling generate","Increase max_seq_len in the model config JSON — bounded by the KV-cache memory your GPUs have","For interactive mode, use the built-in /clear command to reset accumulated history","Pre-check: if len(prompt_tokens[0]) > model.max_seq_len, trim or fail with your own message before calling generate"],"exampleFix":"# before\ncompletion_tokens = generate(model, [prompt_tokens], max_new_tokens, eos_id, temperature)\n\n# after\nmax_prompt = model.max_seq_len - max_new_tokens\nif len(prompt_tokens) > max_prompt:\n    prompt_tokens = prompt_tokens[:max_prompt]  # or raise a clear error\ncompletion_tokens = generate(model, [prompt_tokens], max_new_tokens, eos_id, temperature)","handlingStrategy":"validation","validationCode":"def safe_generate(model, prompt_tokens, max_new_tokens, eos_id, temperature=1.0):\n    longest = max(len(t) for t in prompt_tokens)\n    if longest + 1 > model.max_seq_len:  # leave room for >=1 new token\n        raise ValueError(\n            f\"prompt len {longest} + 1 new token exceeds max_seq_len {model.max_seq_len}; \"\n            f\"truncate the prompt or raise max_seq_len in the config\"\n        )\n    return generate(model, prompt_tokens, max_new_tokens, eos_id, temperature)","typeGuard":"def prompt_fits(prompt_len: int, max_new_tokens: int, max_seq_len: int) -> bool:\n    return prompt_len + max(1, max_new_tokens) <= max_seq_len","tryCatchPattern":"try:\n    tokens = generate(model, prompts, max_new_tokens, eos_id, temp)\nexcept AssertionError as e:\n    if \"max sequence length\" in str(e):\n        messages = messages[-2:]  # drop oldest turns and retry once\n        prompts = [tokenizer.apply_chat_template(messages, add_generation_prompt=True)]\n        tokens = generate(model, prompts, max_new_tokens, eos_id, temp)\n    else:\n        raise","preventionTips":["Trim chat history in interactive mode (use /clear) once it approaches max_seq_len","Budget prompt + max_new_tokens against max_seq_len before calling generate","Remember apply_chat_template adds special tokens on top of your visible text length"],"tags":["generation","context-length","kv-cache","validation","deepseek"],"backgroundTag":null,"analyzedSha":"9b4e9788e4a3a731f7567338ed15d3ec549ce03b","analyzedAt":"2026-08-14T19:02:32.748Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}