{"record":{"id":"10454be6bafdb74a","repo":"invoke-ai/InvokeAI","slug":"prompt-has-num-text-tokens-tokens-exceeds-max-t","errorCode":null,"errorMessage":"prompt has {num_text_tokens} tokens, exceeds max_text_tokens={max_text_tokens}","messagePattern":"prompt has (.+?) tokens, exceeds max_text_tokens=(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/ideogram4/text_encoding.py","lineNumber":48,"sourceCode":"    \"\"\"Encode a single prompt into Ideogram 4 conditioning features.\n\n    Returns a ``(num_text_tokens, 53248)`` float32 tensor (on the encoder's device;\n    the caller is responsible for moving it to CPU for storage).\n    \"\"\"\n    # Importing here keeps module import cheap and tolerant of transformers versions\n    # that lay out the masking utilities differently.\n    from transformers.masking_utils import create_causal_mask\n\n    device = next(text_encoder.parameters()).device\n\n    # Chat-format and tokenize, matching Ideogram4Pipeline._tokenize.\n    messages = [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": prompt}]}]\n    text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\n    encoded = tokenizer(text, return_tensors=\"pt\", add_special_tokens=False)\n    token_ids = encoded[\"input_ids\"].to(device)  # (1, L)\n    num_text_tokens = int(token_ids.shape[1])\n    if num_text_tokens > max_text_tokens:\n        raise ValueError(f\"prompt has {num_text_tokens} tokens, exceeds max_text_tokens={max_text_tokens}\")\n\n    # Text-only sequence: every position is a real LLM token.\n    attention_mask = torch.ones((1, num_text_tokens), dtype=torch.long, device=device)\n    pos_2d = torch.arange(num_text_tokens, device=device)[None, :]  # (1, L)\n\n    language_model = text_encoder.language_model\n    inputs_embeds = language_model.embed_tokens(token_ids)\n\n    position_ids_4d = pos_2d[None, ...].expand(4, 1, num_text_tokens)\n    text_position_ids = position_ids_4d[0]  # (1, L)\n    mrope_position_ids = position_ids_4d[1:]  # (3, 1, L)\n\n    causal_mask = create_causal_mask(\n        config=language_model.config,\n        inputs_embeds=inputs_embeds,\n        attention_mask=attention_mask,\n        past_key_values=None,\n        position_ids=text_position_ids,","sourceCodeStart":30,"sourceCodeEnd":66,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/ideogram4/text_encoding.py#L30-L66","documentation":"encode_qwen3vl_prompt tokenizes the prompt through the Qwen3-VL chat template and refuses prompts whose token count exceeds max_text_tokens, since longer sequences would exceed the model's context/positional budget for text conditioning.","triggerScenarios":"Calling invoke with a very long prompt (or prompt+chat-template overhead) whose tokenized length exceeds max_text_tokens.","commonSituations":"Long descriptive prompts pasted from elsewhere, chat template adding system/generation tokens the user didn't count, small max_text_tokens configured for speed, non-English text tokenizing to more tokens.","solutions":["Shorten the prompt until it fits within max_text_tokens","Increase max_text_tokens if the model/config allows a larger text budget","Count tokens first with the tokenizer and trim before calling","Reduce chat-template overhead by checking what apply_chat_template adds"],"exampleFix":"# before\nencode_qwen3vl_prompt(tokenizer, very_long_prompt, max_text_tokens=256)\n# after\nn = len(tokenizer(very_long_prompt).input_ids)\nassert n <= 256, f\"trim prompt: {n} tokens\"\nencode_qwen3vl_prompt(tokenizer, trim(very_long_prompt, 256), max_text_tokens=256)","handlingStrategy":"validation","validationCode":"encoded = tokenizer(prompt, add_special_tokens=False)\nif len(encoded.input_ids) + CHAT_TEMPLATE_OVERHEAD > max_text_tokens:\n    prompt = truncate_prompt_to_tokens(prompt, max_text_tokens)","typeGuard":null,"tryCatchPattern":"try:\n    result = encode_qwen3vl_prompt(tokenizer, prompt, max_text_tokens, device)\nexcept ValueError as e:\n    if \"exceeds max_text_tokens\" in str(e):\n        prompt = truncate_prompt_to_tokens(prompt, max_text_tokens)\n        result = encode_qwen3vl_prompt(tokenizer, prompt, max_text_tokens, device)\n    else:\n        raise","preventionTips":["Count tokens with the tokenizer before invoking, not by character length","Account for chat-template overhead tokens in your budget","Enforce prompt length limits in the UI/API layer"],"tags":["validation","tokenizer","prompt-length"],"backgroundTag":"prompt-too-long","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}