oobabooga/textgen · error · InvalidRequestError

max_tokens must be greater than 0.

Error message

max_tokens must be greater than 0.

What it means

In the chat completions path, after process_parameters maps 'max_tokens' to 'max_new_tokens', a value that is set and <= 0 raises InvalidRequestError (400, param='max_tokens'). Note this is stricter than the text-completions path (errors 13/14): the chat endpoint does not allow 0 even for logprob-only requests; None means auto (512 + auto_max_new_tokens).

Source

Thrown at modules/api/completions.py:581

    generate_params.update({
        'mode': body['mode'],
        'name1': name1,
        'name2': name2,
        'context': context,
        'greeting': greeting,
        'user_bio': user_bio,
        'instruction_template_str': instruction_template_str,
        'custom_system_message': custom_system_message,
        'chat_template_str': chat_template_str,
        'chat-instruct_command': chat_instruct_command,
        'tools': tools,
        'history': history,
        'stream': stream
    })

    max_tokens = generate_params['max_new_tokens']
    if max_tokens is not None and max_tokens <= 0:
        raise InvalidRequestError(message="max_tokens must be greater than 0.", param="max_tokens")

    if max_tokens is None:
        generate_params['max_new_tokens'] = 512
        generate_params['auto_max_new_tokens'] = True

    requested_model = generate_params.pop('model')
    logprob_proc = generate_params.pop('logprob_proc', None)
    if logprob_proc:
        logprob_proc.token_alternatives_history.clear()
    chat_logprobs_offset = [0]  # mutable for closure access in streaming

    def chat_streaming_chunk(content=None, chunk_tool_calls=None, include_role=False, reasoning_content=None):
        # begin streaming
        delta = {}
        if include_role:
            delta['role'] = 'assistant'
            delta['refusal'] = None
        if content is not None:

View on GitHub (pinned to ed888c71f2)

Solutions

  1. Send a positive integer: {"max_tokens": 512}.
  2. To request auto sizing, omit 'max_tokens' entirely (server then defaults to 512 + auto_max_new_tokens).
  3. Clamp computed budgets client-side: max(1, budget - used) or drop the field when <= 0.
  4. If you wanted a 0-token logprob-only call, use the /v1/completions endpoint with logprobs instead.

Example fix

# before
body = {"model": m, "messages": msgs, "max_tokens": max(0, budget - used)}

# after
body = {"model": m, "messages": msgs}
if budget - used > 0:
    body["max_tokens"] = budget - used
Defensive patterns

Strategy: validation

Validate before calling

def resolve_max_tokens(remaining: int | None):
    if remaining is None or remaining <= 0:
        return None  # omit -> server auto sizing (512 + auto_max_new_tokens)
    return remaining

body = {'model': m, 'messages': msgs}
mt = resolve_max_tokens(budget - used)
if mt is not None:
    body['max_tokens'] = mt

Type guard

def is_valid_chat_max_tokens(v) -> bool:
    return v is None or (isinstance(v, int) and not isinstance(v, bool) and v > 0)

Try / catch

try:
    resp = client.chat.completions.create(model=m, messages=msgs, max_tokens=mt)
except openai.BadRequestError as e:
    if 'max_tokens must be greater than 0' in str(e):
        resp = client.chat.completions.create(model=m, messages=msgs)  # omit: auto
    else:
        raise

Prevention

When it happens

Trigger: POST /v1/chat/completions with {"max_tokens": 0} or a negative value. Compute-bound code doing max_tokens = budget - used and sending it when the budget is exhausted (0 or negative) is a classic trigger.

Common situations: Token-budget managers passing 0 when the context is full; sending -1 as an 'unlimited' convention from other inference servers (this backend treats None, not -1, as auto); test payloads with 0.

Related errors

AI-assisted analysis of oobabooga/textgen@ed888c71f2 (2026-08-15). Data as JSON: /api/errors/6428d334caa0295c. Report an issue: GitHub.