headroomlabs-ai/headroom · error · ProxyError::Upstream

upstream request failed: {0}

Error message

upstream request failed: {0}

What it means

Generic reqwest failure while performing the upstream HTTP request. IntoResponse specializes it: is_timeout() maps to 504 Gateway Timeout, is_connect() maps to 502 Bad Gateway, everything else is 502 with the raw error text. It is the proxy's workhorse network error.

Source

Thrown at crates/headroom-proxy/src/error.rs:9

//! Error types for the proxy.

use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use thiserror::Error;

#[derive(Debug, Error)]
pub enum ProxyError {
    #[error("upstream request failed: {0}")]
    Upstream(#[from] reqwest::Error),

    #[error("invalid upstream URL: {0}")]
    InvalidUpstream(String),

    #[error("invalid header: {0}")]
    InvalidHeader(String),

    #[error("websocket error: {0}")]
    WebSocket(String),

    #[error("io error: {0}")]
    Io(#[from] std::io::Error),

    /// PR-A8 / P5-59: request body exceeded the configured cap. RFC 7231
    /// §6.5.11: 413 Payload Too Large. Previously surfaced as
    /// `InvalidHeader` (400) which mis-classified an oversize body as a
    /// header parse error; clients with retry-on-413 logic broke.

View on GitHub (pinned to 322425c43b)

Solutions

1. Classify first: the HTTP status the client sees (504 vs 502) already tells you timeout vs connect/other — check which you got.
2. For 504: raise the proxy's upstream request timeout above worst-case generation time, or make streaming requests so the connection stays hot.
3. For 502 on connect: verify upstream host/port/DNS and network egress from the proxy host (curl the same endpoint).
4. Enable retries with backoff for idempotent calls; check the upstream provider's status page.

match client.execute(req).await {
    Err(e) if e.is_timeout() => retry_with_backoff().await,        // 504 path
    Err(e) if e.is_connect() => fail_fast(BadGateway, e),           // 502 path
    Err(e) => fail_fast(BadGateway, e),
    Ok(resp) => handle(resp).await,
}

When it happens

Trigger: Any reqwest::Error bubbled via #[from]: connection refused/reset to the upstream LLM endpoint, DNS resolution failure, TLS handshake error, request timeout exceeding the configured client timeout, body read interrupted mid-response.

Common situations: Upstream provider outage or throttling that manifests as resets; wrong upstream URL/port in config; egress blocked by firewall/proxy; client timeout set lower than slow LLM generation latency, causing spurious 504s on long completions.

Related errors

• invalid upstream URL: {0}
• bedrock_upstream_exception
• metrics response build error: {e}
• Copilot token exchange failed with HTTP {exc.code}: {body}
• `{' '.join(cmd)}` produced no output for {idle_cap}s. Check

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/bb7f5d888c35379d. Report an issue: GitHub.