databendlabs/databend · error · Unexpected

{desc}

Error message

{desc}

What it means

Built by `to_opendal_unexpected_error`, the common mapping of a failed `reqwest` send in `http_util::Client::fetch` into an `opendal::Error` of kind Unexpected. The error carries the operation name, the target URL, a human description, and the original reqwest error as its source; it is marked temporary when the failure looks transient (connection reset, timeout, 5xx-style conditions).

Solutions

1. Inspect `err.source()`/source chain and the `url` context to see whether it's DNS, connect, TLS, or timeout.
2. If the error is marked temporary, retry with backoff — the client already classified it as transient.
3. Verify network reachability to the endpoint from the node (curl the URL, check firewall/security-group rules).
4. Check TLS certificate validity on the endpoint if the source error is a handshake failure.
5. Correct the endpoint/URL in storage config if it points at a wrong host or port.

Example fix

// before: unreachable endpoint
let client = HttpClient::new();
let resp = client.fetch("https://storage.internal:9443", ...).await?;
// after: correct, reachable endpoint
let resp = client.fetch("https://s3.us-east-1.amazonaws.com", ...).await?;

fn endpoint_reachable(url: &str) -> bool {
    std::net::TcpStream::connect((
        url::Url::parse(url).ok().and_then(|u| u.host_str().map(String::from))?.as_str(),
        url::Url::parse(url).ok().and_then(|u| u.port_or_known_default()).unwrap_or(443),
    )).is_ok()
}

fn is_temporary_opendal_error(e: &opendal::Error) -> bool {
    e.is_temporary()
}

match client.fetch(url, range).await {
    Ok(r) => Ok(r),
    Err(e) if e.is_temporary() => retry_with_backoff(3, || client.fetch(url, range)).await,
    Err(e) => Err(anyhow!("request to {url} failed permanently: {e}")),
}

When it happens

Trigger: Any `fetch` where the underlying reqwest request fails at the transport level: DNS failure, connection refused/reset, TLS handshake error, request timeout, or the body send/redirect machinery erroring — anything surfaced as `reqwest::Error`.

Common situations: Storage endpoint unreachable from the node (network partition, firewall, security group), DNS misconfiguration, expired/mismatched TLS certificates, or transient upstream outages during heavy load.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• {err}
• Failed to create client
• bandsave load databend meta data failed
• Failed to get or create operator
• Unsupported storage type

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/3f56b9f5c6b602ee. Report an issue: GitHub.

Appendix: source

Thrown at src/common/storage/src/http_client.rs:313

        mem::swap(hr.headers_mut().unwrap(), resp.headers_mut());

        let bs = HttpBody::new(
            resp.bytes_stream()
                .try_filter(|v| future::ready(!v.is_empty()))
                .map_ok(Buffer::from)
                .map_err(move |err| {
                    to_opendal_unexpected_error(err, &uri, "read data from http response")
                }),
            content_length,
        );

        let resp = hr.body(bs).expect("response must build succeed");
        Ok(resp)
    }
}

fn to_opendal_unexpected_error(err: reqwest::Error, uri: &http::Uri, desc: &str) -> opendal::Error {
    let mut oe = opendal::Error::new(opendal::ErrorKind::Unexpected, desc)
        .with_operation("http_util::Client::send")
        .with_context("url", uri.to_string());
    if is_temporary_error(&err) {
        oe = oe.set_temporary();
    }
    oe = oe.set_source(err);
    oe
}

#[inline]
fn is_temporary_error(err: &reqwest::Error) -> bool {
    // error sending request
    err.is_request() ||
        // request or response body error
        err.is_body() ||
        // error decoding response body, for example, connection reset.
        err.is_decode()
}

View on GitHub (pinned to 288d84d76e)