{"record":{"id":"f4d75fd96371f762","repo":"oobabooga/textgen","slug":"api-batched-generation-not-yet-supported","errorCode":null,"errorMessage":"API Batched generation not yet supported.","messagePattern":"API Batched generation not yet supported\\.","errorType":"exception","errorClass":"InvalidRequestError","httpStatus":400,"severity":"error","filePath":"modules/api/completions.py","lineNumber":977,"sourceCode":"            \"usage\": {\n                \"prompt_tokens\": total_prompt_token_count,\n                \"completion_tokens\": total_completion_token_count,\n                \"total_tokens\": total_prompt_token_count + total_completion_token_count\n            }\n        }\n\n        yield resp\n    else:\n        prompt = body[prompt_str]\n        if isinstance(prompt, list):\n            if prompt and isinstance(prompt[0], int):\n                try:\n                    encoder = tiktoken.encoding_for_model(requested_model)\n                    prompt = encoder.decode(prompt)\n                except KeyError:\n                    prompt = decode(prompt)[0]\n            else:\n                raise InvalidRequestError(message=\"API Batched generation not yet supported.\", param=prompt_str)\n\n        prefix = prompt if echo else ''\n        prompt_input_ids = encode(prompt)\n        token_count = len(prompt_input_ids[0])\n\n        # Check if usage should be included in streaming chunks per OpenAI spec\n        stream_options = body.get('stream_options')\n        include_usage = bool(stream_options) and bool(stream_options.get('include_usage') if isinstance(stream_options, dict) else getattr(stream_options, 'include_usage', False))\n        cmpl_logprobs_offset = [0]  # mutable for closure access in streaming\n\n        def text_streaming_chunk(content):\n            # begin streaming\n            if logprob_proc:\n                chunk_logprobs = format_completion_logprobs(_dict_to_logprob_entries(logprob_proc.token_alternatives))\n            elif shared.args.loader in ('llama.cpp', 'ExLlamav3'):\n                entries, cmpl_logprobs_offset[0] = _get_raw_logprob_entries(cmpl_logprobs_offset[0])\n                chunk_logprobs = format_completion_logprobs(entries) if entries else None\n            else:","sourceCodeStart":959,"sourceCodeEnd":995,"githubUrl":"https://github.com/oobabooga/textgen/blob/ed888c71f221df552750e1834b3654abab8ae345/modules/api/completions.py#L959-L995","documentation":"In the text completions path, if 'prompt' is a list it is only supported when the first element is an int (a token-ID list, which is decoded via tiktoken or the model tokenizer). A list of strings (OpenAI's batch-of-prompts format) raises InvalidRequestError (400, param=prompt_str) because batched generation is not implemented in the streaming/non-streaming completion handler.","triggerScenarios":"POST /v1/completions with {\"prompt\": [\"Hello\", \"World\"]} (multiple string prompts in one request). Note {\"prompt\": [\"single string\"]} still fails: the first element is a str, not an int.","commonSituations":"Porting code that used OpenAI's native batched prompt arrays for throughput; sending a single prompt wrapped in a list by a serialization layer; token-ID lists accidentally converted to strings.","solutions":["Send prompts one per request: iterate client-side and issue a request per prompt string.","If you have token IDs, send them as a list of ints (that path is supported and decoded).","Parallelize with concurrent requests rather than relying on server-side batching.","For a single prompt, pass the bare string, not a list."],"exampleFix":"# before\nresp = requests.post(url, json={\"model\": m, \"prompt\": [\"a\", \"b\"]})\n\n# after\nresps = [requests.post(url, json={\"model\": m, \"prompt\": p}) for p in [\"a\", \"b\"]]","handlingStrategy":"fallback","validationCode":"def is_supported_prompt(p) -> bool:\n    return isinstance(p, str) or (isinstance(p, list) and p and isinstance(p[0], int))","typeGuard":"def is_token_id_prompt(p) -> bool:\n    return isinstance(p, list) and bool(p) and all(isinstance(x, int) for x in p)","tryCatchPattern":"try:\n    resp = client.completions.create(model=m, prompt=prompts)\nexcept openai.BadRequestError as e:\n    if 'Batched generation not yet supported' in str(e):\n        resps = [client.completions.create(model=m, prompt=p) for p in prompts]  # fallback: fan out\n    else:\n        raise","preventionTips":["Never send a list of string prompts; loop and issue one request per prompt.","Wrap a single prompt as a bare string, not a one-element list.","Token-ID lists must be ints, not numeric strings."],"tags":["openai-api","completions","batching","validation"],"backgroundTag":null,"analyzedSha":"ed888c71f221df552750e1834b3654abab8ae345","analyzedAt":"2026-08-15T05:24:21.000Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}