BerriAI/litellm · error · DatabricksException

OAuth M2M token request failed

Error message

OAuth M2M token request failed: {e}

What it means

During Databricks OAuth machine-to-machine auth, LiteLLM posts to the workspace token endpoint (derived from api_base) with grant_type=client_credentials using DATABRICKS_CLIENT_ID/SECRET. The requests.post call itself raised RequestException — DNS failure, connection refused, TLS error, or the hardcoded 30s timeout — re-raised as DatabricksException(500) with the underlying exception text.

Solutions

  1. From the same host/network, curl https://<workspace-host>/oidc/v1/token — if that fails, fix DNS/proxy/egress before touching code
  2. Correct DATABRICKS_API_BASE to the real workspace host (the token URL is derived by stripping /serving-endpoints)
  3. Set HTTPS_PROXY/HTTP_PROXY if a corporate proxy is required for outbound traffic
  4. If latency-driven, reduce network hops or run closer to the workspace region
Defensive patterns

Strategy: retry

Validate before calling

import socket, os
from urllib.parse import urlparse

def token_endpoint_reachable(timeout=5) -> bool:
    host = urlparse(os.getenv('DATABRICKS_API_BASE', '')).hostname
    if not host:
        return False
    try:
        socket.create_connection((host, 443), timeout=timeout).close()
        return True
    except OSError:
        return False

Try / catch

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
def call_databricks():
    try:
        return litellm.completion(model='databricks/my-ep', messages=msgs)
    except Exception as e:
        if 'OAuth M2M token request failed' in str(e) and getattr(e, 'status_code', 0) >= 500:
            raise Retryable(e)  # network-level: DNS/proxy/timeout
        raise

Prevention

When it happens

Trigger: M2M auth path taken (client_id + client_secret + api_base set) while the workspace host is unreachable: mistyped DATABRICKS_API_BASE producing a bad token URL, no egress to *.azuredatabricks.net/*.databricks.com, VPN-only workspace, corporate proxy unconfigured for requests, or slow handshakes exceeding 30s.

Common situations: K8s pod without egress rules allowing the Databricks host; DNS not resolving the workspace URL; HTTPS_PROXY not set in the container; cross-region latency causing the fixed timeout to fire.

Related errors

AI-assisted analysis of BerriAI/litellm@77b7c6c40c (2026-08-18). Data as JSON: /api/errors/5ca8816a7877f8c9. Report an issue: GitHub.

Appendix: source

Thrown at litellm/llms/databricks/common_utils.py:246

        workspace_url = api_base.rstrip("/")
        if "/serving-endpoints" in workspace_url:
            workspace_url = workspace_url.replace("/serving-endpoints", "")

        token_url: Final = f"{workspace_url}/oidc/v1/token"

        try:
            response: Final = requests.post(
                token_url,
                data={
                    "grant_type": "client_credentials",
                    "scope": "all-apis",
                },
                auth=(client_id, client_secret),
                headers={"Content-Type": "application/x-www-form-urlencoded"},
                timeout=30,
            )
        except requests.RequestException as e:
            raise DatabricksException(
                status_code=500,
                message=f"OAuth M2M token request failed: {e}",
            )

        if response.status_code != 200:
            raise DatabricksException(
                status_code=response.status_code,
                message=f"OAuth M2M token request failed: {response.text}",
            )

        token_data: Final = response.json()
        return token_data["access_token"]

    def _get_databricks_credentials(
        self, api_key: str | None, api_base: str | None, headers: dict | None
    ) -> tuple[str, dict]:
        """
        Get Databricks credentials using the Databricks SDK.

View on GitHub (pinned to 77b7c6c40c)