{"record":{"id":"670d14b8e1fb66be","repo":"neondatabase/neon","slug":"failed-to-delete-objects","errorCode":null,"errorMessage":"Failed to delete {}/{} objects","messagePattern":"Failed to delete (.+?)/(.+?) objects","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"libs/remote_storage/src/gcs_bucket.rs","lineNumber":583,"sourceCode":"                    } else {\n                        None\n                    }\n                })\n                .collect();\n\n            if !errors.is_empty() {\n                // Report 10 of them like S3\n                const LOG_UP_TO_N_ERRORS: usize = 10;\n                for (id, code) in errors.iter().take(LOG_UP_TO_N_ERRORS) {\n                    tracing::warn!(\n                        \"DeleteObjects key {} failed with code: {}\",\n                        delete_objects_status.get(id).unwrap(),\n                        code\n                    );\n                }\n\n                return Err(anyhow::anyhow!(\n                    \"Failed to delete {}/{} objects\",\n                    errors.len(),\n                    chunk.len(),\n                ));\n            }\n        }\n\n        Ok(())\n    }\n\n    async fn head_object(\n        &self,\n        key: String,\n        cancel: &CancellationToken,\n    ) -> Result<GCSObject, DownloadError> {\n        let kind = RequestKind::Head;\n        let _permit = self.permit(kind, cancel).await?;\n\n        let encoded_path: String = url::form_urlencoded::byte_serialize(key.as_bytes()).collect();","sourceCodeStart":565,"sourceCodeEnd":601,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/remote_storage/src/gcs_bucket.rs#L565-L601","documentation":"The GCS batch delete response reported one or more per-object failures (the JSON API batch endpoint returns per-key results with error codes). The code logs up to 10 failing keys with their codes as warnings, then fails the whole chunk with 'Failed to delete {failed}/{total} objects'.","triggerScenarios":"delete_objects where any object in the batch returns an error code — service account missing storage.objects.delete, precondition failures, or transient per-key errors while other keys in the same batch succeeded.","commonSituations":"IAM role lacking delete permission (all keys fail); GC racing with concurrent writers/modifiers (some keys fail); partial network failures mid-batch.","solutions":["Read the preceding warn logs — they contain the exact failing keys and GCS error codes","Verify the identity has storage.objects.delete on the bucket","Retry the failed subset with backoff — deleting an already-deleted object is a success in GCS, so retries converge","If racing with writers, re-run the deletion later as a sweep rather than failing the whole operation"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Partial batch failure: inspect the warned codes, retry the chunk; already-deleted keys converge.\nasync fn delete_with_retry(storage: &Arc<GenericRemoteStorage>, oids: &[String], cancel: &CancellationToken) -> anyhow::Result<()> {\n    let mut pending: Vec<String> = oids.to_vec();\n    for attempt in 0..3 {\n        if pending.is_empty() { return Ok(()); }\n        match storage.delete_objects(&pending, cancel).await {\n            Ok(()) => return Ok(()),\n            Err(e) if format!(\"{e:#}\").contains(\"Failed to delete\") && attempt < 2 => {\n                tracing::warn!(\"batch delete partial failure (attempt {attempt}), retrying {} keys\", pending.len());\n                tokio::time::sleep(Duration::from_secs(1)).await;\n            }\n            Err(e) => return Err(e),\n        }\n    }\n    unreachable!()\n}","preventionTips":["Grant storage.objects.delete to every identity that runs GC/deletion","Watch the per-key warn logs — they identify exactly which objects and codes failed","Design GC sweeps to tolerate and re-sweep partial failures instead of failing the whole pass"],"tags":["gcs","google-cloud-storage","batch-delete","permissions","iam"],"backgroundTag":"batch-delete-failed","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}