{"record":{"id":"b86ede518594348b","repo":"BerriAI/litellm","slug":"datadogllmobs-unexpected-response-status-code","errorCode":null,"errorMessage":"DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}","messagePattern":"DataDogLLMObs: Unexpected response - status_code: (.+?), text: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"warning","filePath":"litellm/integrations/datadog/datadog_llm_obs.py","lineNumber":205,"sourceCode":"            try:\n                verbose_logger.debug(\"payload %s\", safe_dumps(payload))\n            except Exception as debug_error:\n                verbose_logger.debug(\"payload serialization failed: %s\", str(debug_error))\n\n            json_payload: Final = safe_dumps(payload)\n\n            headers: Final = {\"Content-Type\": \"application/json\"}\n            if self.DD_API_KEY:\n                headers[\"DD-API-KEY\"] = self.DD_API_KEY\n\n            response: Final = await self.async_client.post(\n                url=self.intake_url,\n                content=json_payload,\n                headers=headers,\n            )\n\n            if response.status_code != 202:\n                raise Exception(\n                    f\"DataDogLLMObs: Unexpected response - status_code: {response.status_code}, text: {response.text}\"\n                )\n\n            if self.is_mock_mode:\n                verbose_logger.debug(\"[DATADOG MOCK] Batch of %s events successfully mocked\", len(self.log_queue))\n            else:\n                verbose_logger.debug(\"DataDogLLMObs: Successfully sent batch - status_code: %s\", response.status_code)\n            self.log_queue.clear()\n        except httpx.HTTPStatusError as e:\n            verbose_logger.exception(\"DataDogLLMObs: Error sending batch - %s\", e.response.text)\n        except Exception as e:\n            verbose_logger.exception(\"DataDogLLMObs: Error sending batch - %s\", e)\n\n    def create_llm_obs_payload(self, kwargs: dict, start_time: datetime, end_time: datetime) -> LLMObsPayload:\n        standard_logging_payload: Final[StandardLoggingPayload | None] = kwargs.get(\"standard_logging_object\")\n        if standard_logging_payload is None:\n            raise Exception(\"DataDogLLMObs: standard_logging_object is not set\")\n","sourceCodeStart":187,"sourceCodeEnd":223,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/integrations/datadog/datadog_llm_obs.py#L187-L223","documentation":"DataDogLLMObsLogger flushes queued LLM-Obs spans by POSTing a batch to the trace intake and expects HTTP 202. A non-202 status raises this Exception with the status code and body; the surrounding handlers catch it and log 'DataDogLLMObs: Error sending batch', clear-or-retry semantics aside, so traces in that batch are lost but the proxy keeps serving.","triggerScenarios":"Direct-API mode POST to https://api.<DD_SITE>/api/intake/llm-obs/v1/trace/spans returns 403 (bad/revoked DD_API_KEY), 404 (wrong DD_SITE for the org), 400 (malformed span payload), or 429 (rate limit). Agent mode can also produce 5xx if the local agent is unhealthy or LITELLM_DD_LLM_OBS_PORT (default 8126) is wrong.","commonSituations":"LLM-Obs key without LLM-Obs entitlement; site/key mismatch; local Datadog agent without the trace/APM endpoint enabled on port 8126; large bursts of spans during load tests tripping 429s.","solutions":["Read the logged status/text: 403 -> verify the API key and its LLM-Obs entitlement, 404 -> fix DD_SITE, 429 -> reduce flush volume/batch size","For agent mode, confirm the agent listens on LITELLM_DD_LLM_OBS_PORT (default 8126) and APM/trace intake is enabled","Send one manual span via curl to the intake URL to isolate key/site issues from payload issues"],"exampleFix":"# before\n# local agent, wrong port assumption\n# LITELLM_DD_AGENT_HOST=localhost (agent APM on 10518 disabled for traces)\n\n# after\nexport LITELLM_DD_AGENT_HOST=localhost\nexport LITELLM_DD_LLM_OBS_PORT=8126  # trace agent port","handlingStrategy":"try-catch","validationCode":"import httpx, os\n\nurl = f\"https://api.{os.getenv('DD_SITE')}/api/intake/llm-obs/v1/trace/spans\"\nr = httpx.post(url, headers={\"DD-API-KEY\": os.environ[\"DD_API_KEY\"]}, json={\"data\": []})\nprint(r.status_code, r.text)  # expect 202","typeGuard":null,"tryCatchPattern":"# litellm catches and logs this per batch; guard by monitoring:\n# alert on 'DataDogLLMObs: Error sending batch' in logs;\n# correlate status text (403 key / 404 site / 429 rate) to the fix","preventionTips":["Verify the API key has the LLM Observability entitlement, not just logs","For agent mode, confirm the trace agent port (LITELLM_DD_LLM_OBS_PORT, default 8126) is open","Watch trace completeness in Datadog after deploys — dropped batches are otherwise silent"],"tags":["datadog","llm-observability","http","batching"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}