BerriAI/litellm · warning · Exception

DataDogLLMObs: Unexpected response - status_code

Error message

DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}

What it means

DataDogLLMObsLogger.flush path: async_send_batch POSTs the queued LLM-obs spans to the /api/intake/llm-obs/v1/trace/spans endpoint and requires HTTP 202. Unlike the sync datadog.py path there is no raise_for_status() first, so every non-202 — including 400/401/403/429/5xx — surfaces as this generic Exception. It is caught by the trailing `except Exception` and only logged; importantly log_queue.clear() only runs on success, so failed batches stay queued and are retried on the next flush.

Solutions

  1. Turn on verbose logging and read the logged status_code/text — 401/403 means key, 404 means site/URL, 413 means batch too large
  2. Verify DD_API_KEY and DD_SITE, or in agent mode verify LITELLM_DD_AGENT_HOST and LITELLM_DD_LLM_OBS_PORT point at the trace agent (default 8126)
  3. Test the intake directly with curl POST expecting 202
  4. If 413/429, reduce batch size / flush interval so payloads shrink

Example fix

# before
export LITELLM_DD_AGENT_HOST=localhost   # but trace agent not on 8126 -> persistent non-202

# after
export LITELLM_DD_AGENT_HOST=localhost
export LITELLM_DD_LLM_OBS_PORT=8126      # match the actual trace-agent port
# or direct API:
export DD_API_KEY=<key> DD_SITE=us5.datadoghq.com
Defensive patterns

Strategy: validation

Validate before calling

import os, httpx

def llm_obs_intake_reachable() -> bool:
    if os.getenv("LITELLM_DD_AGENT_HOST"):
        host = os.getenv("LITELLM_DD_AGENT_HOST")
        port = os.getenv("LITELLM_DD_LLM_OBS_PORT", "8126")
        url = f"http://{host}:{port}/api/intake/llm-obs/v1/trace/spans"
    else:
        if not (os.getenv("DD_API_KEY") and os.getenv("DD_SITE")):
            return False
        url = f"https://api.{os.getenv('DD_SITE')}/api/intake/llm-obs/v1/trace/spans"
    try:
        r = httpx.post(url, headers={"DD-API-KEY": os.getenv("DD_API_KEY", "")}, json={"data": {}})
        return r.status_code in (202, 400)  # 400 = reachable+authed but empty body
    except httpx.HTTPError:
        return False

Try / catch

// The flush loop already catches and logs; callers add nothing.
// Guard instead against unbounded retry: watch the queue on failure.

Prevention

When it happens

Trigger: Invalid or revoked DD_API_KEY (403), wrong DD_SITE (404/DNS issues), oversized batch hitting 413, malformed span payload rejected with 400, rate limiting (429), or an agent-mode URL pointing at a port running the wrong agent (logs agent instead of trace agent).

Common situations: Key rotated in Datadog but not in the proxy deployment; LITELLM_DD_LLM_OBS_PORT left default 8126 while the trace agent listens elsewhere; large batch_size with big payloads; intermittent retries spamming logs because the queue never drains.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/b86ede518594348b. Report an issue: GitHub.

Appendix: source

Thrown at litellm/integrations/datadog/datadog_llm_obs.py:205

            try:
                verbose_logger.debug("payload %s", safe_dumps(payload))
            except Exception as debug_error:
                verbose_logger.debug("payload serialization failed: %s", str(debug_error))

            json_payload: Final = safe_dumps(payload)

            headers: Final = {"Content-Type": "application/json"}
            if self.DD_API_KEY:
                headers["DD-API-KEY"] = self.DD_API_KEY

            response: Final = await self.async_client.post(
                url=self.intake_url,
                content=json_payload,
                headers=headers,
            )

            if response.status_code != 202:
                raise Exception(
                    f"DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}"
                )

            if self.is_mock_mode:
                verbose_logger.debug("[DATADOG MOCK] Batch of %s events successfully mocked", len(self.log_queue))
            else:
                verbose_logger.debug("DataDogLLMObs: Successfully sent batch - status_code: %s", response.status_code)
            self.log_queue.clear()
        except httpx.HTTPStatusError as e:
            verbose_logger.exception("DataDogLLMObs: Error sending batch - %s", e.response.text)
        except Exception as e:
            verbose_logger.exception("DataDogLLMObs: Error sending batch - %s", e)

    def create_llm_obs_payload(self, kwargs: dict, start_time: datetime, end_time: datetime) -> LLMObsPayload:
        standard_logging_payload: Final[StandardLoggingPayload | None] = kwargs.get("standard_logging_object")
        if standard_logging_payload is None:
            raise Exception("DataDogLLMObs: standard_logging_object is not set")

View on GitHub (pinned to 77b7c6c40c)