BerriAI/litellm · error · HuggingFaceError

response is not in expected format - {completion_response}

Error message

response is not in expected format - {completion_response}

What it means

Raised when task='text-generation-inference' and the parsed completion response is not a list of dicts each containing 'generated_text'. The TGI response format is [{"generated_text": "...", ...}], and anything else (dict, string, missing key) means the endpoint did not return TGI-style output.

Source

Thrown at litellm/llms/huggingface/embedding/transformation.py:388

        task: hf_tasks | None,
        optional_params: dict,
        encoding: Any,
        messages: list[AllMessageValues],
        model: str,
    ):
        if task is None:
            task = "text-generation-inference"  # default to tgi

        if task == "conversational":
            if len(completion_response["generated_text"]) > 0:
                model_response.choices[0].message.content = completion_response["generated_text"]
        elif task == "text-generation-inference":
            if (
                not isinstance(completion_response, list)
                or not isinstance(completion_response[0], dict)
                or "generated_text" not in completion_response[0]
            ):
                raise HuggingFaceError(
                    status_code=422,
                    message=f"response is not in expected format - {completion_response}",
                    headers=None,
                )

            if len(completion_response[0]["generated_text"]) > 0:
                model_response.choices[0].message.content = output_parser(completion_response[0]["generated_text"])
            ## GETTING LOGPROBS + FINISH REASON
            if "details" in completion_response[0] and "tokens" in completion_response[0]["details"]:
                model_response.choices[0].finish_reason = completion_response[0]["details"]["finish_reason"]
                sum_logprob = 0
                for token in completion_response[0]["details"]["tokens"]:
                    if token["logprob"] is not None:
                        sum_logprob += token["logprob"]
                setattr(model_response.choices[0].message, "_logprob", sum_logprob)
            if "best_of" in optional_params and optional_params["best_of"] > 1:
                if "details" in completion_response[0] and "best_of_sequences" in completion_response[0]["details"]:
                    choices_list: Final = []

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Check the echoed completion_response in the message to see what the server actually returned.
2. If the endpoint returns a plain string or object, use task='conversational' or 'text-generation' to match the schema.
3. If api_base points at a non-HF server (e.g. OpenAI-compatible vLLM), switch the provider instead of forcing huggingface + TGI task.

Example fix

# before
litellm.completion(model='huggingface/my-model', api_base='http://localhost:8000', task='text-generation-inference', messages=messages)

# after  # vLLM / OpenAI-compatible server
litellm.completion(model='openai/my-model', api_base='http://localhost:8000/v1', messages=messages)

def is_tgi_response(obj: object) -> bool:
    return (isinstance(obj, list) and len(obj) > 0
            and isinstance(obj[0], dict) and 'generated_text' in obj[0])

try:
    resp = litellm.completion(model=model, messages=messages, task='text-generation-inference')
except litellm.llms.huggingface.common_utils.HuggingFaceError as e:
    if 'not in expected format' in str(e):
        raise RuntimeError(f'endpoint {api_base} does not speak TGI; check task/provider') from e
    raise

When it happens

Trigger: Pointing task='text-generation-inference' at a non-TGI endpoint (e.g. a conversational endpoint returning {"generated_text": ...} directly, a JSON error object, or an HTML error page that happened to parse as JSON).

Common situations: Self-hosted server (vLLM, plain text-generation server, custom FastAPI) behind an api_base that returns a different schema; HF inference endpoints migrated from TGI to the new router API returning different JSON.

Related errors

• Original Response received: {raw_response.text}
• Invalid hf task - {task}. Valid formats - {hf_tasks}.
• {completion_response[error]}
• {raw_response.text}
• litellm.JSONSchemaValidationError: model={model}, returned a

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/1a0eff4003460590. Report an issue: GitHub.