BerriAI/litellm · warning · Exception

DataDogLLMObs: Unexpected response - status_code: {response.

Error message

DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}

What it means

DataDogLLMObsLogger flushes queued LLM-Obs spans by POSTing a batch to the trace intake and expects HTTP 202. A non-202 status raises this Exception with the status code and body; the surrounding handlers catch it and log 'DataDogLLMObs: Error sending batch', clear-or-retry semantics aside, so traces in that batch are lost but the proxy keeps serving.

Source

Thrown at litellm/integrations/datadog/datadog_llm_obs.py:205

            try:
                verbose_logger.debug("payload %s", safe_dumps(payload))
            except Exception as debug_error:
                verbose_logger.debug("payload serialization failed: %s", str(debug_error))

            json_payload: Final = safe_dumps(payload)

            headers: Final = {"Content-Type": "application/json"}
            if self.DD_API_KEY:
                headers["DD-API-KEY"] = self.DD_API_KEY

            response: Final = await self.async_client.post(
                url=self.intake_url,
                content=json_payload,
                headers=headers,
            )

            if response.status_code != 202:
                raise Exception(
                    f"DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}"
                )

            if self.is_mock_mode:
                verbose_logger.debug("[DATADOG MOCK] Batch of %s events successfully mocked", len(self.log_queue))
            else:
                verbose_logger.debug("DataDogLLMObs: Successfully sent batch - status_code: %s", response.status_code)
            self.log_queue.clear()
        except httpx.HTTPStatusError as e:
            verbose_logger.exception("DataDogLLMObs: Error sending batch - %s", e.response.text)
        except Exception as e:
            verbose_logger.exception("DataDogLLMObs: Error sending batch - %s", e)

    def create_llm_obs_payload(self, kwargs: dict, start_time: datetime, end_time: datetime) -> LLMObsPayload:
        standard_logging_payload: Final[StandardLoggingPayload | None] = kwargs.get("standard_logging_object")
        if standard_logging_payload is None:
            raise Exception("DataDogLLMObs: standard_logging_object is not set")

View on GitHub (pinned to 6c2dcb801b)

Solutions

Read the logged status/text: 403 -> verify the API key and its LLM-Obs entitlement, 404 -> fix DD_SITE, 429 -> reduce flush volume/batch size
For agent mode, confirm the agent listens on LITELLM_DD_LLM_OBS_PORT (default 8126) and APM/trace intake is enabled
Send one manual span via curl to the intake URL to isolate key/site issues from payload issues

Example fix

# before
# local agent, wrong port assumption
# LITELLM_DD_AGENT_HOST=localhost (agent APM on 10518 disabled for traces)

# after
export LITELLM_DD_AGENT_HOST=localhost
export LITELLM_DD_LLM_OBS_PORT=8126  # trace agent port

Defensive patterns

Strategy: try-catch

Validate before calling

import httpx, os

url = f"https://api.{os.getenv('DD_SITE')}/api/intake/llm-obs/v1/trace/spans"
r = httpx.post(url, headers={"DD-API-KEY": os.environ["DD_API_KEY"]}, json={"data": []})
print(r.status_code, r.text)  # expect 202

Try / catch

# litellm catches and logs this per batch; guard by monitoring:
# alert on 'DataDogLLMObs: Error sending batch' in logs;
# correlate status text (403 key / 404 site / 429 rate) to the fix

Prevention

Verify the API key has the LLM Observability entitlement, not just logs
For agent mode, confirm the trace agent port (LITELLM_DD_LLM_OBS_PORT, default 8126) is open
Watch trace completeness in Datadog after deploys — dropped batches are otherwise silent

When it happens

Trigger: Direct-API mode POST to https://api.<DD_SITE>/api/intake/llm-obs/v1/trace/spans returns 403 (bad/revoked DD_API_KEY), 404 (wrong DD_SITE for the org), 400 (malformed span payload), or 429 (rate limit). Agent mode can also produce 5xx if the local agent is unhealthy or LITELLM_DD_LLM_OBS_PORT (default 8126) is wrong.

Common situations: LLM-Obs key without LLM-Obs entitlement; site/key mismatch; local Datadog agent without the trace/APM endpoint enabled on port 8126; large bursts of spans during load tests tripping 429s.

Related errors

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/b86ede518594348b. Report an issue: GitHub.