{"record":{"id":"1a0eff4003460590","repo":"BerriAI/litellm","slug":"response-is-not-in-expected-format-completion-r","errorCode":null,"errorMessage":"response is not in expected format - {completion_response}","messagePattern":"response is not in expected format - (.+?)","errorType":"http","errorClass":"HuggingFaceError","httpStatus":422,"severity":"error","filePath":"litellm/llms/huggingface/embedding/transformation.py","lineNumber":388,"sourceCode":"        task: hf_tasks | None,\n        optional_params: dict,\n        encoding: Any,\n        messages: list[AllMessageValues],\n        model: str,\n    ):\n        if task is None:\n            task = \"text-generation-inference\"  # default to tgi\n\n        if task == \"conversational\":\n            if len(completion_response[\"generated_text\"]) > 0:\n                model_response.choices[0].message.content = completion_response[\"generated_text\"]\n        elif task == \"text-generation-inference\":\n            if (\n                not isinstance(completion_response, list)\n                or not isinstance(completion_response[0], dict)\n                or \"generated_text\" not in completion_response[0]\n            ):\n                raise HuggingFaceError(\n                    status_code=422,\n                    message=f\"response is not in expected format - {completion_response}\",\n                    headers=None,\n                )\n\n            if len(completion_response[0][\"generated_text\"]) > 0:\n                model_response.choices[0].message.content = output_parser(completion_response[0][\"generated_text\"])\n            ## GETTING LOGPROBS + FINISH REASON\n            if \"details\" in completion_response[0] and \"tokens\" in completion_response[0][\"details\"]:\n                model_response.choices[0].finish_reason = completion_response[0][\"details\"][\"finish_reason\"]\n                sum_logprob = 0\n                for token in completion_response[0][\"details\"][\"tokens\"]:\n                    if token[\"logprob\"] is not None:\n                        sum_logprob += token[\"logprob\"]\n                setattr(model_response.choices[0].message, \"_logprob\", sum_logprob)\n            if \"best_of\" in optional_params and optional_params[\"best_of\"] > 1:\n                if \"details\" in completion_response[0] and \"best_of_sequences\" in completion_response[0][\"details\"]:\n                    choices_list: Final = []","sourceCodeStart":370,"sourceCodeEnd":406,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/llms/huggingface/embedding/transformation.py#L370-L406","documentation":"Raised when task='text-generation-inference' and the parsed completion response is not a list of dicts each containing 'generated_text'. The TGI response format is [{\"generated_text\": \"...\", ...}], and anything else (dict, string, missing key) means the endpoint did not return TGI-style output.","triggerScenarios":"Pointing task='text-generation-inference' at a non-TGI endpoint (e.g. a conversational endpoint returning {\"generated_text\": ...} directly, a JSON error object, or an HTML error page that happened to parse as JSON).","commonSituations":"Self-hosted server (vLLM, plain text-generation server, custom FastAPI) behind an api_base that returns a different schema; HF inference endpoints migrated from TGI to the new router API returning different JSON.","solutions":["Check the echoed completion_response in the message to see what the server actually returned.","If the endpoint returns a plain string or object, use task='conversational' or 'text-generation' to match the schema.","If api_base points at a non-HF server (e.g. OpenAI-compatible vLLM), switch the provider instead of forcing huggingface + TGI task."],"exampleFix":"# before\nlitellm.completion(model='huggingface/my-model', api_base='http://localhost:8000', task='text-generation-inference', messages=messages)\n\n# after  # vLLM / OpenAI-compatible server\nlitellm.completion(model='openai/my-model', api_base='http://localhost:8000/v1', messages=messages)","handlingStrategy":"type-guard","validationCode":null,"typeGuard":"def is_tgi_response(obj: object) -> bool:\n    return (isinstance(obj, list) and len(obj) > 0\n            and isinstance(obj[0], dict) and 'generated_text' in obj[0])","tryCatchPattern":"try:\n    resp = litellm.completion(model=model, messages=messages, task='text-generation-inference')\nexcept litellm.llms.huggingface.common_utils.HuggingFaceError as e:\n    if 'not in expected format' in str(e):\n        raise RuntimeError(f'endpoint {api_base} does not speak TGI; check task/provider') from e\n    raise","preventionTips":["Smoke-test one completion per endpoint at startup to detect schema mismatches early","Match task= to the actual server software (TGI vs conversational vs OpenAI-compatible)"],"tags":["huggingface","completion","response-format","tgi"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}