{"record":{"id":"b86ede518594348b","repo":"BerriAI/litellm","slug":"datadogllmobs-unexpected-response-status-code","errorCode":null,"errorMessage":"DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}","messagePattern":"DataDogLLMObs: Unexpected response - status_code: (.+?), text: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"warning","filePath":"litellm/integrations/datadog/datadog_llm_obs.py","lineNumber":205,"sourceCode":"            try:\n                verbose_logger.debug(\"payload %s\", safe_dumps(payload))\n            except Exception as debug_error:\n                verbose_logger.debug(\"payload serialization failed: %s\", str(debug_error))\n\n            json_payload: Final = safe_dumps(payload)\n\n            headers: Final = {\"Content-Type\": \"application/json\"}\n            if self.DD_API_KEY:\n                headers[\"DD-API-KEY\"] = self.DD_API_KEY\n\n            response: Final = await self.async_client.post(\n                url=self.intake_url,\n                content=json_payload,\n                headers=headers,\n            )\n\n            if response.status_code != 202:\n                raise Exception(\n                    f\"DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}\"\n                )\n\n            if self.is_mock_mode:\n                verbose_logger.debug(\"[DATADOG MOCK] Batch of %s events successfully mocked\", len(self.log_queue))\n            else:\n                verbose_logger.debug(\"DataDogLLMObs: Successfully sent batch - status_code: %s\", response.status_code)\n            self.log_queue.clear()\n        except httpx.HTTPStatusError as e:\n            verbose_logger.exception(\"DataDogLLMObs: Error sending batch - %s\", e.response.text)\n        except Exception as e:\n            verbose_logger.exception(\"DataDogLLMObs: Error sending batch - %s\", e)\n\n    def create_llm_obs_payload(self, kwargs: dict, start_time: datetime, end_time: datetime) -> LLMObsPayload:\n        standard_logging_payload: Final[StandardLoggingPayload | None] = kwargs.get(\"standard_logging_object\")\n        if standard_logging_payload is None:\n            raise Exception(\"DataDogLLMObs: standard_logging_object is not set\")\n","sourceCodeStart":187,"sourceCodeEnd":223,"githubUrl":"https://github.com/BerriAI/litellm/blob/77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8/litellm/integrations/datadog/datadog_llm_obs.py#L187-L223","documentation":"DataDogLLMObsLogger.flush path: async_send_batch POSTs the queued LLM-obs spans to the /api/intake/llm-obs/v1/trace/spans endpoint and requires HTTP 202. Unlike the sync datadog.py path there is no raise_for_status() first, so every non-202 — including 400/401/403/429/5xx — surfaces as this generic Exception. It is caught by the trailing `except Exception` and only logged; importantly log_queue.clear() only runs on success, so failed batches stay queued and are retried on the next flush.","triggerScenarios":"Invalid or revoked DD_API_KEY (403), wrong DD_SITE (404/DNS issues), oversized batch hitting 413, malformed span payload rejected with 400, rate limiting (429), or an agent-mode URL pointing at a port running the wrong agent (logs agent instead of trace agent).","commonSituations":"Key rotated in Datadog but not in the proxy deployment; LITELLM_DD_LLM_OBS_PORT left default 8126 while the trace agent listens elsewhere; large batch_size with big payloads; intermittent retries spamming logs because the queue never drains.","solutions":["Turn on verbose logging and read the logged status_code/text — 401/403 means key, 404 means site/URL, 413 means batch too large","Verify DD_API_KEY and DD_SITE, or in agent mode verify LITELLM_DD_AGENT_HOST and LITELLM_DD_LLM_OBS_PORT point at the trace agent (default 8126)","Test the intake directly with curl POST expecting 202","If 413/429, reduce batch size / flush interval so payloads shrink"],"exampleFix":"# before\nexport LITELLM_DD_AGENT_HOST=localhost   # but trace agent not on 8126 -> persistent non-202\n\n# after\nexport LITELLM_DD_AGENT_HOST=localhost\nexport LITELLM_DD_LLM_OBS_PORT=8126      # match the actual trace-agent port\n# or direct API:\nexport DD_API_KEY=<key> DD_SITE=us5.datadoghq.com","handlingStrategy":"validation","validationCode":"import os, httpx\n\ndef llm_obs_intake_reachable() -> bool:\n    if os.getenv(\"LITELLM_DD_AGENT_HOST\"):\n        host = os.getenv(\"LITELLM_DD_AGENT_HOST\")\n        port = os.getenv(\"LITELLM_DD_LLM_OBS_PORT\", \"8126\")\n        url = f\"http://{host}:{port}/api/intake/llm-obs/v1/trace/spans\"\n    else:\n        if not (os.getenv(\"DD_API_KEY\") and os.getenv(\"DD_SITE\")):\n            return False\n        url = f\"https://api.{os.getenv('DD_SITE')}/api/intake/llm-obs/v1/trace/spans\"\n    try:\n        r = httpx.post(url, headers={\"DD-API-KEY\": os.getenv(\"DD_API_KEY\", \"\")}, json={\"data\": {}})\n        return r.status_code in (202, 400)  # 400 = reachable+authed but empty body\n    except httpx.HTTPError:\n        return False","typeGuard":null,"tryCatchPattern":"// The flush loop already catches and logs; callers add nothing.\n// Guard instead against unbounded retry: watch the queue on failure.","preventionTips":["Health-check the llm-obs intake (202 path) from the same network as the proxy","Keep batch_size modest so 413/429 responses can't wedge the queue","Alert when 'DataDogLLMObs: Error sending batch' appears repeatedly — the queue retries forever until it succeeds"],"tags":["datadog","llm-obs","http-status","async","batching","telemetry-loss"],"backgroundTag":"unexpected-http-status-code","analyzedSha":"77b7c6c40c0c5aa5fbcb1d6a1825ac39ca8829b8","analyzedAt":"2026-08-18T11:44:31.656Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}