{"record":{"id":"9a02c1bc9f344c5e","repo":"deepseek-ai/DeepSeek-V3","slug":"number-of-prompts-exceeds-maximum-batch-size-ar","errorCode":null,"errorMessage":"Number of prompts exceeds maximum batch size (${args.max_batch_size})","messagePattern":"Number of prompts exceeds maximum batch size \\((.+?)\\)","errorType":"validation","errorClass":"AssertionError","httpStatus":null,"severity":"error","filePath":"inference/generate.py","lineNumber":148,"sourceCode":"            else:\n                objects = [None]\n                dist.broadcast_object_list(objects, 0)\n                prompt = objects[0]\n            if prompt == \"/exit\":\n                break\n            elif prompt == \"/clear\":\n                messages.clear()\n                continue\n            messages.append({\"role\": \"user\", \"content\": prompt})\n            prompt_tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)\n            completion_tokens = generate(model, [prompt_tokens], max_new_tokens, tokenizer.eos_token_id, temperature)\n            completion = tokenizer.decode(completion_tokens[0], skip_special_tokens=True)\n            print(completion)\n            messages.append({\"role\": \"assistant\", \"content\": completion})\n    else:\n        with open(input_file) as f:\n            prompts = [line.strip() for line in f.readlines()]\n        assert len(prompts) <= args.max_batch_size, f\"Number of prompts exceeds maximum batch size ({args.max_batch_size})\"\n        prompt_tokens = [tokenizer.apply_chat_template([{\"role\": \"user\", \"content\": prompt}], add_generation_prompt=True) for prompt in prompts]\n        completion_tokens = generate(model, prompt_tokens, max_new_tokens, tokenizer.eos_token_id, temperature)\n        completions = tokenizer.batch_decode(completion_tokens, skip_special_tokens=True)\n        for prompt, completion in zip(prompts, completions):\n            print(\"Prompt:\", prompt)\n            print(\"Completion:\", completion)\n            print()\n\n    if world_size > 1:\n        dist.destroy_process_group()\n\n\nif __name__ == \"__main__\":\n    \"\"\"\n    Command-line interface for distributed text generation.\n\n    Arguments:\n        --ckpt-path (str): Path to the model checkpoint directory.","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/generate.py#L130-L166","documentation":"Thrown in the batched file mode of main() (inference/generate.py:148): every line of --input-file becomes one sequence in a single batched generate() call, and the batch dimension of the preallocated token tensor cannot exceed args.max_batch_size. This is the mainline script's assert; interactive mode (one prompt at a time) never hits it.","triggerScenarios":"Running generate.py with --input-file containing more non-empty lines than --max-batch-size (default in main's call, constrained by GPU KV-cache capacity). Each line is one prompt.","commonSituations":"Feeding a large eval/prompts file (hundreds of lines) while the max batch was set small for memory reasons; forgetting to strip trailing blank lines which still count after .strip() if non-empty; assuming the script auto-chunks batches (it does not).","solutions":["Split the input file into chunks of at most max_batch_size lines and run per chunk","Raise max_batch_size if KV-cache memory allows (it exists to bound memory, so increase only with headroom)","Filter empty lines: prompts = [l for l in (line.strip() for line in f) if l] before counting"],"exampleFix":"# before\nwith open(input_file) as f:\n    prompts = [line.strip() for line in f.readlines()]\n\n# after — chunked processing\nwith open(input_file) as f:\n    prompts = [line.strip() for line in f if line.strip()]\nfor i in range(0, len(prompts), args.max_batch_size):\n    batch = prompts[i:i + args.max_batch_size]\n    toks = [tokenizer.apply_chat_template([{\"role\": \"user\", \"content\": p}], add_generation_prompt=True) for p in batch]\n    outs = generate(model, toks, max_new_tokens, tokenizer.eos_token_id, temperature)\n    for p, o in zip(batch, tokenizer.batch_decode(outs, skip_special_tokens=True)):\n        print(p, '->', o)","handlingStrategy":"validation","validationCode":"with open(input_file) as f:\n    prompts = [line.strip() for line in f if line.strip()]\nif len(prompts) > MAX_BATCH_SIZE:\n    raise ValueError(\n        f\"{len(prompts)} prompts > max_batch_size {MAX_BATCH_SIZE}; \"\n        f\"split the file or raise the batch size\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Chunk input files to max_batch_size lines before launching","Strip empty lines when counting","Remember max_batch_size bounds GPU memory via the preallocated token/KV tensors"],"tags":["batching","generation","cli","validation","deepseek"],"backgroundTag":null,"analyzedSha":"9b4e9788e4a3a731f7567338ed15d3ec549ce03b","analyzedAt":"2026-08-14T19:02:32.748Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}