neondatabase/neon · error · DownloadError

GCS head response contained no response body

Error message

GCS head response contained no response body

What it means

The GCS object metadata (HEAD-style) request returned a non-success status other than 404. Because a metadata response carries no body to include, the code reports a generic 'no response body' message; the real classification is the HTTP status, which the catch-all match arm discards. Auth (401/403), throttling (429/529), and server errors (5xx) all surface identically here.

Source

Thrown at libs/remote_storage/src/gcs_bucket.rs:646

            _ = cancel.cancelled() => return Err(TimeoutOrCancel::Cancel.into()),
        };

        let res = res.map_err(|_e| DownloadError::Timeout)?;

        // do not incl. timeouts as errors in metrics but cancellations
        let started_at = ScopeGuard::into_inner(started_at);
        crate::metrics::BUCKET_METRICS
            .req_seconds
            .observe_elapsed(kind, &res, started_at);

        let data = match res {
            Ok(data) => {
                if !data.status().is_success() {
                    match data.status() {
                        StatusCode::NOT_FOUND => return Err(DownloadError::NotFound),
                        _ => {
                            return Err(DownloadError::Other(anyhow::anyhow!(
                                "GCS head response contained no response body"
                            )));
                        }
                    }
                } else {
                    data
                }
            }
            Err(e) => {
                crate::metrics::BUCKET_METRICS.req_seconds.observe_elapsed(
                    kind,
                    AttemptOutcome::Err,
                    started_at,
                );

                return Err(DownloadError::Other(
                    anyhow::Error::new(e).context("error in HEAD of GCS object"),
                ));
            }

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Capture the actual status before the match discards it — patch or debug-log data.status() so the failure is classifiable
2. Refresh credentials and retry
3. Apply backoff retry for 429/5xx statuses
4. Check IAM: storage.objects.get on the bucket for the identity in use

Example fix

// before: every non-404 failure is an opaque error
match data.status() {
    StatusCode::NOT_FOUND => return Err(DownloadError::NotFound),
    _ => return Err(DownloadError::Other(anyhow::anyhow!("GCS head response contained no response body"))),
}

// after: carry the status so callers can retry intelligently
match data.status() {
    StatusCode::NOT_FOUND => return Err(DownloadError::NotFound),
    status => return Err(DownloadError::Other(anyhow::anyhow!("GCS head failed: {}", status))),
}

// Generic message hides the status: retry transient-looking head failures, surface the rest.
async fn head_with_retry(storage: &Arc<GenericRemoteStorage>, key: String, cancel: &CancellationToken) -> Result<ListingObject, DownloadError> {
    let mut attempt = 0;
    loop {
        attempt += 1;
        match storage.head_object(key.clone(), cancel).await {
            Ok(obj) => return Ok(obj),
            Err(DownloadError::NotFound) => return Err(DownloadError::NotFound),
            Err(DownloadError::Other(e)) if attempt < 3 => {
                tracing::warn!("GCS head failed (attempt {attempt}): {e:#}");
                tokio::time::sleep(Duration::from_millis(300 * attempt as u64)).await;
            }
            Err(e) => return Err(e),
        }
    }
}

When it happens

Trigger: head_object receiving 401/403 (expired token or missing storage.objects.get), 429/529 (rate limiting), or 5xx — anything except 200 and 404.

Common situations: Expired gcp_auth token in a long-lived process; missing IAM permission; GCS throttling under heavy listing/probing load; transient 5xx during GCP incidents.

Related errors

• GCS GET response contained no response body
• GCS PUT error \n\t {:?}
• no items returned
• no 'updated' field
• max keys reached

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/a2232b86814dc4fd. Report an issue: GitHub.