neondatabase/neon · error

GCS PUT error \n\t {:?}

Error message

GCS PUT error \n\t {:?}

What it means

The GCS multipart upload PUT (uploadType=multipart) returned a non-success HTTP status and the whole reqwest::Response is formatted into the error. Any 4xx/5xx lands here: 401/403 (auth or IAM), 400 (malformed multipart body), 429/529 (throttling), 5xx (server). The embedded Response debug output — including the status code — is the key to classifying the failure.

Source

Thrown at libs/remote_storage/src/gcs_bucket.rs:398

        let res = tokio::select! {
            res = upload => res,
            _ = cancel.cancelled() => return Err(TimeoutOrCancel::Cancel.into()),
        };

        // not if let-ing an Ok(inner), since res is not double-Result<>-wrapped with the tokio
        // timeout, observe_elapsed's AttemptedOutcome trait obj expects
        // &Result<reqwest::Response> which &res directly is, and it can handle the Err case.
        let started_at = ScopeGuard::into_inner(started_at);
        crate::metrics::BUCKET_METRICS
            .req_seconds
            .observe_elapsed(kind, &res, started_at);
        
        match res {
            Ok(res) => {
                if !res.status().is_success() {
                    match res.status() {
                        _ => Err(anyhow::anyhow!("GCS PUT error \n\t {:?}", res)),
                    }
                } else {
                    let body = res
                        .text()
                        .await
                        .map_err(|e: reqwest::Error| DownloadError::Other(e.into()))?;

                    let resp: GCSObject = serde_json::from_str(&body)
                        .map_err(|e: serde_json::Error| DownloadError::Other(e.into()))?;

                    if !resp.size.is_some_and(|s| s == fs_size as i64) {
                        // very unlikely
                        return Err(anyhow::anyhow!(
                            "Boundary string from 'multipart/related' HTTP upload occurred in payload"
                        ));
                    };

                    Ok(())

View on GitHub (pinned to 8f60b04da4)

Solutions

  1. Read the embedded Response debug output — the HTTP status classifies the failure (401/403 auth, 400 request, 429/529 throttle, 5xx retry)
  2. Refresh/re-auth the token provider and retry
  3. Verify the service account has storage.objects.create on the target bucket and the bucket name is correct
  4. Retry with backoff on 429/529/5xx; do not retry 4xx other than 429

Example fix

// before: status code swallowed into a generic anyhow error
match res.status() {
    _ => Err(anyhow::anyhow!("GCS PUT error \n\t {:?}", res)),
}

// after: classify and chain the status + body
let status = res.status();
if !status.is_success() {
    let body = res.text().await.unwrap_or_default();
    return Err(anyhow::anyhow!("GCS PUT error: {} body: {}", status, body));
}
Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-flight the token provider so expired credentials surface before the upload starts.
async fn token_ok(provider: &GcpTokenProvider) -> bool {
    provider.token(GCS_SCOPES).await.is_ok()
}

Try / catch

// Parse the embedded Response debug to classify; retry only transient statuses.
match storage.upload(&data, fs_size, &cancel).await {
    Ok(()) => Ok(()),
    Err(e) => {
        let msg = format!("{e:#}");
        let transient = msg.contains("429") || msg.contains("529")
            || msg.contains("500") || msg.contains("503");
        let auth = msg.contains("401") || msg.contains("403");
        if auth {
            refresh_credentials().await?; // then retry once
            storage.upload(&data, fs_size, &cancel).await
        } else if transient {
            backoff_retry_upload(storage, &data, fs_size, &cancel).await
        } else {
            Err(e)
        }
    }
}

Prevention

When it happens

Trigger: put_object with an expired or invalid OAuth token from the gcp_auth token provider; service account lacking storage.objects.create on the bucket; malformed multipart form body; per-bucket rate limits; GCS transient 5xx.

Common situations: Long-running processes whose GCP token expired; wrong bucket IAM binding; bulk uploads hitting rate limits; streaming sources truncating the request body mid-upload.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/9e08f407858d19d7. Report an issue: GitHub.