BerriAI/litellm · error · ValueError

Instances are required for private endpoint

Error message

Instances are required for private endpoint

What it means

Async private-endpoint guard (PredictionServiceAsyncClient path): instances is None, but the await-ed predict call requires instance dicts. Mirrors the sync check and fails fast before any network I/O.

Source

Thrown at litellm/llms/vertex_ai/vertex_ai_non_gemini.py:656

        )
        request_str += f"llm_model = aiplatform.gapic.PredictionServiceAsyncClient(client_options={client_options}, credentials=...)\n"
        endpoint_path: Final = llm_model.endpoint_path(project=vertex_project, location=vertex_location, endpoint=model)
        request_str += f"client.predict(endpoint={endpoint_path}, instances={instances})\n"
        response_obj = await llm_model.predict(
            endpoint=endpoint_path,
            instances=instances,
        )

        response = response_obj.predictions
        completion_response = response[0]
        if isinstance(completion_response, str) and "\nOutput:\n" in completion_response:
            completion_response = completion_response.split("\nOutput:\n", 1)[1]
        if stream:
            response = TextStreamer(completion_response)

    elif mode == "private":
        if instances is None:
            raise ValueError("Instances are required for private endpoint")
        stream = optional_params.pop("stream", None)
        _ = instances[0].pop("stream", None)
        request_str += f"llm_model.predict_async(instances={instances})\n"
        response_obj = await llm_model.predict_async(
            instances=instances,
        )
        response = response_obj.predictions
        completion_response = response[0]
        if isinstance(completion_response, str) and "\nOutput:\n" in completion_response:
            completion_response = completion_response.split("\nOutput:\n", 1)[1]
        if stream:
            response = TextStreamer(completion_response)

    if response is None:
        raise ValueError("Unable to generate response")

    logging_obj.post_call(input=prompt, api_key=None, original_response=response)

View on GitHub (pinned to 77b7c6c40c)

Solutions

1. Include the 'instances' field in the request to the private endpoint.
2. Review the model's serving input contract and supply instances in the required format.

When it happens

Trigger: Thrown at litellm/llms/vertex_ai/vertex_ai_non_gemini.py:656 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/bcdc95463671bada. Report an issue: GitHub.