{"record":{"id":"113e41104d793bd2","repo":"neondatabase/neon","slug":"failed-to-delete-objects-113e41","errorCode":null,"errorMessage":"Failed to delete {}/{} objects","messagePattern":"Failed to delete (.+?)/(.+?) objects","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"libs/remote_storage/src/s3_bucket.rs","lineNumber":400,"sourceCode":"                .inc_by(chunk.len() as u64);\n\n            if let Some(errors) = resp.errors {\n                // Log a bounded number of the errors within the response:\n                // these requests can carry 1000 keys so logging each one\n                // would be too verbose, especially as errors may lead us\n                // to retry repeatedly.\n                const LOG_UP_TO_N_ERRORS: usize = 10;\n                for e in errors.iter().take(LOG_UP_TO_N_ERRORS) {\n                    tracing::warn!(\n                        \"DeleteObjects key {} failed: {}: {}\",\n                        e.key.as_ref().map(Cow::from).unwrap_or(\"\".into()),\n                        e.code.as_ref().map(Cow::from).unwrap_or(\"\".into()),\n                        e.message.as_ref().map(Cow::from).unwrap_or(\"\".into())\n                    );\n                }\n\n                return Err(anyhow::anyhow!(\n                    \"Failed to delete {}/{} objects\",\n                    errors.len(),\n                    chunk.len(),\n                ));\n            }\n        }\n        Ok(())\n    }\n\n    async fn list_versions_with_permit(\n        &self,\n        _permit: &tokio::sync::SemaphorePermit<'_>,\n        prefix: Option<&RemotePath>,\n        mode: ListingMode,\n        max_keys: Option<NonZeroU32>,\n        cancel: &CancellationToken,\n    ) -> Result<crate::VersionListing, DownloadError> {\n        // get the passed prefix or if it is not set use prefix_in_bucket value\n        let prefix = prefix","sourceCodeStart":382,"sourceCodeEnd":418,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/remote_storage/src/s3_bucket.rs#L382-L418","documentation":"delete_objects on S3Bucket chunks keys (up to 1000 per DeleteObjects call); when the batch response contains per-object error entries, the code logs the first 10 (key, code, message) at warn level and fails the whole operation with this summary error. Per-key errors carry S3 codes such as AccessDenied, NoSuchVersion, or InternalError.","triggerScenarios":"Calling delete/delete_objects (e.g. pageserver GC deleting layers) when the IAM role lacks s3:DeleteObject, objects are under object-lock/legal hold, the bucket policy forbids delete, or S3 returns transient InternalError for a subset of keys.","commonSituations":"GC starts failing after an IAM policy change; object lock enabled on the bucket; S3-compatible providers returning partial failures in multi-delete requests.","solutions":["Inspect the preceding 'DeleteObjects key ... failed: <code>: <message>' warn logs to get the per-object S3 error code","Grant s3:DeleteObject on the bucket in the IAM/bucket policy","Check for object lock / legal hold on the affected keys","Retry the operation: transient codes (InternalError) resolve; AccessDenied will not until permissions change"],"exampleFix":"// before: policy lacks delete\n{\"Effect\":\"Allow\",\"Action\":[\"s3:GetObject\",\"s3:PutObject\"],\"Resource\":\"arn:aws:s3:::bucket/*\"}\n// after\n{\"Effect\":\"Allow\",\"Action\":[\"s3:GetObject\",\"s3:PutObject\",\"s3:DeleteObject\"],\"Resource\":\"arn:aws:s3:::bucket/*\"}","handlingStrategy":"retry","validationCode":"use aws_sdk_s3::Client;\n\nasync fn delete_permission_ok(client: &Client, bucket: &str) -> anyhow::Result<()> {\n    // canary single-key delete surfaces IAM problems before a large batch runs\n    client.delete_object().bucket(bucket).key(\"__perm_probe__\").send().await?;\n    Ok(())\n}","typeGuard":"fn is_partial_delete_failure(err: &anyhow::Error) -> bool {\n    err.to_string().starts_with(\"Failed to delete\")\n}","tryCatchPattern":"let mut attempt = 0;\nloop {\n    match storage.delete_objects(&paths, &cancel).await {\n        Ok(()) => break,\n        Err(e) if is_partial_delete_failure(&e) && attempt < 3 => {\n            attempt += 1;\n            tokio::time::sleep(std::time::Duration::from_secs(1 << attempt)).await;\n        }\n        Err(e) => return Err(e.context(\"delete_objects failed permanently\")),\n    }\n}","preventionTips":["Pre-provision IAM with s3:DeleteObject from day one","Parse the per-key 'DeleteObjects key ... failed' warn logs to classify permanent vs transient codes","Keep deletes idempotent so retries are safe"],"tags":["s3","rust","aws-sdk","iam","batch-delete"],"backgroundTag":"s3-batch-delete-partial-failure","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}