oobabooga/textgen · error · InvalidRequestError

API Batched generation not yet supported.

Error message

API Batched generation not yet supported.

What it means

In the text completions path, if 'prompt' is a list it is only supported when the first element is an int (a token-ID list, which is decoded via tiktoken or the model tokenizer). A list of strings (OpenAI's batch-of-prompts format) raises InvalidRequestError (400, param=prompt_str) because batched generation is not implemented in the streaming/non-streaming completion handler.

Source

Thrown at modules/api/completions.py:977

            "usage": {
                "prompt_tokens": total_prompt_token_count,
                "completion_tokens": total_completion_token_count,
                "total_tokens": total_prompt_token_count + total_completion_token_count
            }
        }

        yield resp
    else:
        prompt = body[prompt_str]
        if isinstance(prompt, list):
            if prompt and isinstance(prompt[0], int):
                try:
                    encoder = tiktoken.encoding_for_model(requested_model)
                    prompt = encoder.decode(prompt)
                except KeyError:
                    prompt = decode(prompt)[0]
            else:
                raise InvalidRequestError(message="API Batched generation not yet supported.", param=prompt_str)

        prefix = prompt if echo else ''
        prompt_input_ids = encode(prompt)
        token_count = len(prompt_input_ids[0])

        # Check if usage should be included in streaming chunks per OpenAI spec
        stream_options = body.get('stream_options')
        include_usage = bool(stream_options) and bool(stream_options.get('include_usage') if isinstance(stream_options, dict) else getattr(stream_options, 'include_usage', False))
        cmpl_logprobs_offset = [0]  # mutable for closure access in streaming

        def text_streaming_chunk(content):
            # begin streaming
            if logprob_proc:
                chunk_logprobs = format_completion_logprobs(_dict_to_logprob_entries(logprob_proc.token_alternatives))
            elif shared.args.loader in ('llama.cpp', 'ExLlamav3'):
                entries, cmpl_logprobs_offset[0] = _get_raw_logprob_entries(cmpl_logprobs_offset[0])
                chunk_logprobs = format_completion_logprobs(entries) if entries else None
            else:

View on GitHub (pinned to ed888c71f2)

Solutions

1. Send prompts one per request: iterate client-side and issue a request per prompt string.
2. If you have token IDs, send them as a list of ints (that path is supported and decoded).
3. Parallelize with concurrent requests rather than relying on server-side batching.
4. For a single prompt, pass the bare string, not a list.

Example fix

# before
resp = requests.post(url, json={"model": m, "prompt": ["a", "b"]})

# after
resps = [requests.post(url, json={"model": m, "prompt": p}) for p in ["a", "b"]]

def is_supported_prompt(p) -> bool:
    return isinstance(p, str) or (isinstance(p, list) and p and isinstance(p[0], int))

def is_token_id_prompt(p) -> bool:
    return isinstance(p, list) and bool(p) and all(isinstance(x, int) for x in p)

try:
    resp = client.completions.create(model=m, prompt=prompts)
except openai.BadRequestError as e:
    if 'Batched generation not yet supported' in str(e):
        resps = [client.completions.create(model=m, prompt=p) for p in prompts]  # fallback: fan out
    else:
        raise

When it happens

Trigger: POST /v1/completions with {"prompt": ["Hello", "World"]} (multiple string prompts in one request). Note {"prompt": ["single string"]} still fails: the first element is a str, not an int.

Common situations: Porting code that used OpenAI's native batched prompt arrays for throughput; sending a single prompt wrapped in a list by a serialization layer; token-ID lists accidentally converted to strings.

Related errors

• Missing required input
• max_tokens must be greater than or equal to 0.
• max_tokens is 0 but no logprobs parameter was specified.
• functions is not supported.
• function_call is not supported.

AI-assisted analysis of oobabooga/textgen@ed888c71f2 (2026-08-15). Data as JSON: /api/errors/f4d75fd96371f762. Report an issue: GitHub.