neondatabase/neon · error · DownloadError

download gcs object

Error message

download gcs object

What it means

The catch-all arm after the metadata GET in download(): it fires when the request itself failed at the reqwest transport layer (connection error, DNS failure, TLS error, request build/send error) rather than returning an HTTP response. The message 'download gcs object' is uninformative because the underlying reqwest::Error is dropped instead of being chained into the context.

Source

Thrown at libs/remote_storage/src/gcs_bucket.rs:745

        };

        let resp = match obj_metadata {
            Ok(resp) => {
                if !resp.status().is_success() {
                    match resp.status() {
                        StatusCode::NOT_FOUND => return Err(DownloadError::NotFound),
                        _ => {
                            return Err(DownloadError::Other(anyhow::anyhow!(
                                "GCS GET response contained no response body"
                            )));
                        }
                    }
                } else {
                    resp
                }
            }
            _ => {
                return Err(DownloadError::Other(anyhow::anyhow!("download gcs object")));
            }
        };

        let body = resp
            .text()
            .await
            .map_err(|e: reqwest::Error| DownloadError::Other(e.into()))?;

        let resp: GCSObject = serde_json::from_str(&body)
            .map_err(|e: serde_json::Error| DownloadError::Other(e.into()))?;

        // 2. Byte Stream request
        let mut headers = header::HeaderMap::new();
        let bytes_range = match &request.range {
           Some(s) => header::HeaderValue::from_str(s).unwrap(),
           None => header::HeaderValue::from_static("bytes=0-"),
        };
        

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Check connectivity to storage.googleapis.com from the affected host (curl -v)
2. Retry — transport errors are usually transient
3. Patch or upstream a fix so the error is chained: anyhow::Error::new(e).context(\"download gcs object\") — the cause then becomes visible
4. Enable reqwest/hyper tracing (RUST_LOG=hyper=trace,reqwest=debug) to see the transport-level cause

Example fix

// before: underlying reqwest::Error dropped, only a bare string survives
_ => {
    return Err(DownloadError::Other(anyhow::anyhow!("download gcs object")));
}

// after: chain the cause
Err(e) => {
    return Err(DownloadError::Other(
        anyhow::Error::new(e).context("download gcs object"),
    ));
}

// Cheap connectivity pre-flight when this error recurs in an environment.
async fn gcs_reachable() -> bool {
    tokio::net::TcpStream::connect("storage.googleapis.com:443").await.is_ok()
}

// Transport-level failure: backoff retry; the dropped reqwest::Error means logging context matters.
let download = backoff_retry(
    || storage.download(from, &cancel),
    |e| !matches!(e, DownloadError::NotFound | DownloadError::Cancelled),
    3,
    "gcs download",
    cancel,
).await;

When it happens

Trigger: Network-level failure of the metadata GET: connection refused/reset, DNS resolution failure, TLS handshake error, or proxy misconfiguration — the select! resolves to Err(reqwest::Error) and the arm returns a bare message.

Common situations: Egress/DNS problems in the cluster; HTTP(S)_PROXY misconfiguration; connection resets under heavy load; transient network partitions to storage.googleapis.com.

Related errors

• no items returned
• no 'updated' field
• max keys reached
• GCS PUT error \n\t {:?}
• Boundary string from 'multipart/related' HTTP upload occurre

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/b268ddd9c5bddbca. Report an issue: GitHub.