databendlabs/databend · error · Unexpected

{err}

Error message

{err}

What it means

Raised in `HttpClient::client_for_checked_endpoint` when the `reqwest` client cannot be built from the configured TLS/timeout/keepalive options. The reqwest builder error is converted into an `opendal::Error` with kind Unexpected, meaning the HTTP client itself could not be constructed (not that a request failed).

Solutions

1. Read the wrapped `err.to_string()` — it names the exact reqwest builder failure (TLS cert path, keepalive, etc.).
2. Verify configured TLS certificate/key file paths exist and are readable on the node.
3. Check the storage config (timeout, keepalive, TLS settings) for invalid values and fix them.
4. Ensure the reqwest feature set (native-tls vs rustls) matches the deployed environment; rebuild with the right features.

Example fix

// before: cert path wrong in config
HttpClient::new().with_tls_files("/missing/cert.pem", "/missing/key.pem")?
// after
HttpClient::new().with_tls_files("/etc/databend/cert.pem", "/etc/databend/key.pem")?

fn tls_files_exist(cert: &str, key: &str) -> bool {
    std::path::Path::new(cert).is_file() && std::path::Path::new(key).is_file()
}

fn client_config_ok(cfg: &StorageHttpConfig) -> bool {
    cfg.cert_file.as_deref().map_or(true, |c| std::path::Path::new(c).is_file())
        && cfg.key_file.as_deref().map_or(true, |k| std::path::Path::new(k).is_file())
}

match client.fetch(url, range).await {
    Ok(r) => r,
    Err(e) if e.to_string().contains("Unexpected") => {
        // client construction failed; check TLS config before retrying
        return Err(anyhow!("http client init failed: {e}"));
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling `HttpClient::fetch` where the resolved config produces an invalid client: e.g. an invalid `tcp_keepalive` duration, a TLS config referencing missing certificate/key files or an unsupported TLS backend, or a reqwest feature (like rustls) not compiled in for the requested scheme.

Common situations: Deployment config pointing at certificates that don't exist on the node, mismatched TLS feature flags between crates, or an OS/env where the native-tls backend is unavailable.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• {desc}
• bandsave load databend meta data failed
• Failed to get or create operator
• Unsupported storage type
• {e}

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/8e2609dca6749fde. Report an issue: GitHub.

Appendix: source

Thrown at src/common/storage/src/http_client.rs:156

                // hostname endpoints (closing the DNS-rebinding TOCTOU
                // window) and IP-literal endpoints (closing a 30x
                // bounce-to-internal bypass), since check_url_with_dns now
                // populates resolved_addrs for IP literals as well.
                let mut builder = storage_http_client_builder()
                    .http1_only()
                    .use_native_tls()
                    .dns_resolver(get_global_hickory_resolver())
                    .redirect(reqwest::redirect::Policy::none())
                    .pool_max_idle_per_host(self.pool_max_idle_per_host)
                    .connect_timeout(Duration::from_secs(self.connect_timeout))
                    .resolve_to_addrs(&host, resolved_addrs);

                if self.keepalive != 0 {
                    builder = builder.tcp_keepalive(Duration::from_secs(self.keepalive));
                }

                builder.build().map_err(|err| {
                    opendal::Error::new(opendal::ErrorKind::Unexpected, err.to_string())
                })
            })?
            .clone();

        Ok(client)
    }

    async fn check_endpoint_cached(&self, url: &Url) -> opendal::Result<EndpointUrlCheck> {
        let host = url.host_str().unwrap_or_default().to_ascii_lowercase();
        let port = url.port_or_known_default().unwrap_or_default();
        let key = EndpointCheckCacheKey {
            scheme: url.scheme().to_ascii_lowercase(),
            host,
            port,
        };

        {
            let mut cache = self.checked_endpoints.lock().unwrap();

View on GitHub (pinned to 288d84d76e)