BerriAI/litellm · error · AzureOpenAIError

Failed to parse raw Azure embedding response: {json_error}

Error message

Failed to parse raw Azure embedding response: {json_error}

What it means

When Azure's embedding endpoint returns a body that is not valid JSON, httpx's parse() raises JSONDecodeError, which LiteLLM converts to AzureOpenAIError carrying the upstream status code. The conversion exists deliberately (per the inline comment) so the router sees a status_code for cooldown logic and so httpx connection cleanup runs, preventing connection leaks under load.

Source

Thrown at litellm/llms/azure/azure.py:698

            raw_response: Final = await openai_aclient.embeddings.with_raw_response.create(**data, timeout=timeout)
            headers: Final = dict(raw_response.headers)

            # Convert json.JSONDecodeError to AzureOpenAIError for two critical reasons:
            #
            # 1. ROUTER BEHAVIOR: The router relies on exception.status_code to determine cooldown logic:
            #    - JSONDecodeError has no status_code → router skips cooldown evaluation
            #    - AzureOpenAIError has status_code → router properly evaluates for cooldown
            #
            # 2. CONNECTION CLEANUP: When response.parse() throws JSONDecodeError, the response
            #    body may not be fully consumed, preventing httpx from properly returning the
            #    connection to the pool. By catching the exception and accessing raw_response.status_code,
            #    we trigger httpx's internal cleanup logic. Without this:
            #    - parse() fails → JSONDecodeError bubbles up → httpx never knows response was acknowledged → connection leak
            #    This completely eliminates "Unclosed connection" warnings during high load.
            try:
                response = raw_response.parse()
            except json.JSONDecodeError as json_error:
                raise AzureOpenAIError(
                    status_code=raw_response.status_code or 500,
                    message=f"Failed to parse raw Azure embedding response: {json_error}",
                ) from json_error
            if isinstance(response, str):
                raise AzureOpenAIError(
                    status_code=raw_response.status_code or 500,
                    message=f"Unexpected string response from Azure: {response[:500]}",
                )
            stringified_response: Final = response.model_dump()

            ## LOGGING
            logging_obj.post_call(
                input=input,
                api_key=api_key,
                additional_args={"complete_input_dict": data},
                original_response=stringified_response,
            )
            embedding_response: Final = convert_to_model_response_object(

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Capture the raw body (enable litellm debug/logging) to see what non-JSON payload came back.
2. If a gateway/proxy sits in front of Azure, exempt or fix its error pages for the embeddings path.
3. Retry with backoff — transient malformed responses usually succeed on retry; litellm router cooldown will mark the deployment and reroute.
4. If persistent, compare direct curl calls to the endpoint to confirm Azure itself is not misbehaving.

Example fix

# before
resp = await litellm.aembedding(model='azure/text-embedding-3-large', input=texts)  # raises AzureOpenAIError on bad JSON

# after
from litellm.exceptions import APIError
try:
    resp = await litellm.aembedding(model='azure/text-embedding-3-large', input=texts)
except APIError as e:
    if 'Failed to parse raw Azure embedding response' in str(e):
        await asyncio.sleep(2)
        resp = await litellm.aembedding(model='azure/text-embedding-3-large', input=texts)
    else:
        raise

from litellm.exceptions import APIError

async def embed_with_retry(texts, attempts=3):
    for i in range(attempts):
        try:
            return await litellm.aembedding(model='azure/text-embedding-3-large', input=texts)
        except APIError as e:
            if 'Failed to parse raw Azure embedding response' not in str(e) or i == attempts - 1:
                raise
            await asyncio.sleep(2 ** i)

When it happens

Trigger: Azure or an intermediary (gateway, API management layer, corporate proxy) returns HTML error pages, empty bodies, or truncated JSON for an embeddings request; intermittent 5xx with non-JSON payloads.

Common situations: Traffic routed through Azure API Management with custom error templates; proxy timeouts returning HTML; partial responses under network stress.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Mavvrik FOCUS destination: response missing 'url' field: {re
• embedding_response is not an instance of EmbeddingResponse
• {req_token.text}
• api_base is None. Please set AZURE_AI_API_BASE or dynamicall
• api_key is None. Please set AZURE_AI_API_KEY or dynamically

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/7090bb30ac22824a. Report an issue: GitHub.