deepseek-ai/DeepSeek-V3 · error · AssertionError

Number of prompts exceeds maximum batch size (${args.max_bat

Error message

Number of prompts exceeds maximum batch size (${args.max_batch_size})

What it means

Thrown in the batched file mode of main() (inference/generate.py:148): every line of --input-file becomes one sequence in a single batched generate() call, and the batch dimension of the preallocated token tensor cannot exceed args.max_batch_size. This is the mainline script's assert; interactive mode (one prompt at a time) never hits it.

Source

Thrown at inference/generate.py:148

            else:
                objects = [None]
                dist.broadcast_object_list(objects, 0)
                prompt = objects[0]
            if prompt == "/exit":
                break
            elif prompt == "/clear":
                messages.clear()
                continue
            messages.append({"role": "user", "content": prompt})
            prompt_tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
            completion_tokens = generate(model, [prompt_tokens], max_new_tokens, tokenizer.eos_token_id, temperature)
            completion = tokenizer.decode(completion_tokens[0], skip_special_tokens=True)
            print(completion)
            messages.append({"role": "assistant", "content": completion})
    else:
        with open(input_file) as f:
            prompts = [line.strip() for line in f.readlines()]
        assert len(prompts) <= args.max_batch_size, f"Number of prompts exceeds maximum batch size ({args.max_batch_size})"
        prompt_tokens = [tokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True) for prompt in prompts]
        completion_tokens = generate(model, prompt_tokens, max_new_tokens, tokenizer.eos_token_id, temperature)
        completions = tokenizer.batch_decode(completion_tokens, skip_special_tokens=True)
        for prompt, completion in zip(prompts, completions):
            print("Prompt:", prompt)
            print("Completion:", completion)
            print()

    if world_size > 1:
        dist.destroy_process_group()


if __name__ == "__main__":
    """
    Command-line interface for distributed text generation.

    Arguments:
        --ckpt-path (str): Path to the model checkpoint directory.

View on GitHub (pinned to 9b4e9788e4)

Solutions

Split the input file into chunks of at most max_batch_size lines and run per chunk
Raise max_batch_size if KV-cache memory allows (it exists to bound memory, so increase only with headroom)
Filter empty lines: prompts = [l for l in (line.strip() for line in f) if l] before counting

Example fix

# before
with open(input_file) as f:
    prompts = [line.strip() for line in f.readlines()]

# after — chunked processing
with open(input_file) as f:
    prompts = [line.strip() for line in f if line.strip()]
for i in range(0, len(prompts), args.max_batch_size):
    batch = prompts[i:i + args.max_batch_size]
    toks = [tokenizer.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True) for p in batch]
    outs = generate(model, toks, max_new_tokens, tokenizer.eos_token_id, temperature)
    for p, o in zip(batch, tokenizer.batch_decode(outs, skip_special_tokens=True)):
        print(p, '->', o)

Defensive patterns

Strategy: validation

Validate before calling

with open(input_file) as f:
    prompts = [line.strip() for line in f if line.strip()]
if len(prompts) > MAX_BATCH_SIZE:
    raise ValueError(
        f"{len(prompts)} prompts > max_batch_size {MAX_BATCH_SIZE}; "
        f"split the file or raise the batch size"
    )

Prevention

Chunk input files to max_batch_size lines before launching
Strip empty lines when counting
Remember max_batch_size bounds GPU memory via the preallocated token/KV tensors

When it happens

Trigger: Running generate.py with --input-file containing more non-empty lines than --max-batch-size (default in main's call, constrained by GPU KV-cache capacity). Each line is one prompt.

Common situations: Feeding a large eval/prompts file (hundreds of lines) while the max batch was set small for memory reasons; forgetting to strip trailing blank lines which still count after .strip() if non-empty; assuming the script auto-chunks batches (it does not).

Related errors

AI-assisted analysis of deepseek-ai/DeepSeek-V3@9b4e9788e4 (2026-08-14). Data as JSON: /api/errors/9a02c1bc9f344c5e. Report an issue: GitHub.