{"record":{"id":"9e08f407858d19d7","repo":"neondatabase/neon","slug":"gcs-put-error-n-t","errorCode":null,"errorMessage":"GCS PUT error \\n\\t {:?}","messagePattern":"GCS PUT error \\\\n\\\\t (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"libs/remote_storage/src/gcs_bucket.rs","lineNumber":398,"sourceCode":"\n        let res = tokio::select! {\n            res = upload => res,\n            _ = cancel.cancelled() => return Err(TimeoutOrCancel::Cancel.into()),\n        };\n\n        // not if let-ing an Ok(inner), since res is not double-Result<>-wrapped with the tokio\n        // timeout, observe_elapsed's AttemptedOutcome trait obj expects\n        // &Result<reqwest::Response> which &res directly is, and it can handle the Err case.\n        let started_at = ScopeGuard::into_inner(started_at);\n        crate::metrics::BUCKET_METRICS\n            .req_seconds\n            .observe_elapsed(kind, &res, started_at);\n        \n        match res {\n            Ok(res) => {\n                if !res.status().is_success() {\n                    match res.status() {\n                        _ => Err(anyhow::anyhow!(\"GCS PUT error \\n\\t {:?}\", res)),\n                    }\n                } else {\n                    let body = res\n                        .text()\n                        .await\n                        .map_err(|e: reqwest::Error| DownloadError::Other(e.into()))?;\n\n                    let resp: GCSObject = serde_json::from_str(&body)\n                        .map_err(|e: serde_json::Error| DownloadError::Other(e.into()))?;\n\n                    if !resp.size.is_some_and(|s| s == fs_size as i64) {\n                        // very unlikely\n                        return Err(anyhow::anyhow!(\n                            \"Boundary string from 'multipart/related' HTTP upload occurred in payload\"\n                        ));\n                    };\n\n                    Ok(())","sourceCodeStart":380,"sourceCodeEnd":416,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/remote_storage/src/gcs_bucket.rs#L380-L416","documentation":"The GCS multipart upload PUT (uploadType=multipart) returned a non-success HTTP status and the whole reqwest::Response is formatted into the error. Any 4xx/5xx lands here: 401/403 (auth or IAM), 400 (malformed multipart body), 429/529 (throttling), 5xx (server). The embedded Response debug output — including the status code — is the key to classifying the failure.","triggerScenarios":"put_object with an expired or invalid OAuth token from the gcp_auth token provider; service account lacking storage.objects.create on the bucket; malformed multipart form body; per-bucket rate limits; GCS transient 5xx.","commonSituations":"Long-running processes whose GCP token expired; wrong bucket IAM binding; bulk uploads hitting rate limits; streaming sources truncating the request body mid-upload.","solutions":["Read the embedded Response debug output — the HTTP status classifies the failure (401/403 auth, 400 request, 429/529 throttle, 5xx retry)","Refresh/re-auth the token provider and retry","Verify the service account has storage.objects.create on the target bucket and the bucket name is correct","Retry with backoff on 429/529/5xx; do not retry 4xx other than 429"],"exampleFix":"// before: status code swallowed into a generic anyhow error\nmatch res.status() {\n    _ => Err(anyhow::anyhow!(\"GCS PUT error \\n\\t {:?}\", res)),\n}\n\n// after: classify and chain the status + body\nlet status = res.status();\nif !status.is_success() {\n    let body = res.text().await.unwrap_or_default();\n    return Err(anyhow::anyhow!(\"GCS PUT error: {} body: {}\", status, body));\n}","handlingStrategy":"try-catch","validationCode":"// Pre-flight the token provider so expired credentials surface before the upload starts.\nasync fn token_ok(provider: &GcpTokenProvider) -> bool {\n    provider.token(GCS_SCOPES).await.is_ok()\n}","typeGuard":null,"tryCatchPattern":"// Parse the embedded Response debug to classify; retry only transient statuses.\nmatch storage.upload(&data, fs_size, &cancel).await {\n    Ok(()) => Ok(()),\n    Err(e) => {\n        let msg = format!(\"{e:#}\");\n        let transient = msg.contains(\"429\") || msg.contains(\"529\")\n            || msg.contains(\"500\") || msg.contains(\"503\");\n        let auth = msg.contains(\"401\") || msg.contains(\"403\");\n        if auth {\n            refresh_credentials().await?; // then retry once\n            storage.upload(&data, fs_size, &cancel).await\n        } else if transient {\n            backoff_retry_upload(storage, &data, fs_size, &cancel).await\n        } else {\n            Err(e)\n        }\n    }\n}","preventionTips":["Refresh GCP tokens proactively in long-running processes before expiry","Verify storage.objects.create IAM on every bucket at deploy time via config checks","Always surface the HTTP status in upload errors — a raw Response dump makes classification grep-able"],"tags":["gcs","google-cloud-storage","upload","http-status","auth"],"backgroundTag":"http-upload-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}