neondatabase/neon · error

Failed to delete {}/{} objects

Error message

Failed to delete {}/{} objects

What it means

delete_objects on S3Bucket chunks keys (up to 1000 per DeleteObjects call); when the batch response contains per-object error entries, the code logs the first 10 (key, code, message) at warn level and fails the whole operation with this summary error. Per-key errors carry S3 codes such as AccessDenied, NoSuchVersion, or InternalError.

Source

Thrown at libs/remote_storage/src/s3_bucket.rs:400

                .inc_by(chunk.len() as u64);

            if let Some(errors) = resp.errors {
                // Log a bounded number of the errors within the response:
                // these requests can carry 1000 keys so logging each one
                // would be too verbose, especially as errors may lead us
                // to retry repeatedly.
                const LOG_UP_TO_N_ERRORS: usize = 10;
                for e in errors.iter().take(LOG_UP_TO_N_ERRORS) {
                    tracing::warn!(
                        "DeleteObjects key {} failed: {}: {}",
                        e.key.as_ref().map(Cow::from).unwrap_or("".into()),
                        e.code.as_ref().map(Cow::from).unwrap_or("".into()),
                        e.message.as_ref().map(Cow::from).unwrap_or("".into())
                    );
                }

                return Err(anyhow::anyhow!(
                    "Failed to delete {}/{} objects",
                    errors.len(),
                    chunk.len(),
                ));
            }
        }
        Ok(())
    }

    async fn list_versions_with_permit(
        &self,
        _permit: &tokio::sync::SemaphorePermit<'_>,
        prefix: Option<&RemotePath>,
        mode: ListingMode,
        max_keys: Option<NonZeroU32>,
        cancel: &CancellationToken,
    ) -> Result<crate::VersionListing, DownloadError> {
        // get the passed prefix or if it is not set use prefix_in_bucket value
        let prefix = prefix

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Inspect the preceding 'DeleteObjects key ... failed: <code>: <message>' warn logs to get the per-object S3 error code
2. Grant s3:DeleteObject on the bucket in the IAM/bucket policy
3. Check for object lock / legal hold on the affected keys
4. Retry the operation: transient codes (InternalError) resolve; AccessDenied will not until permissions change

Example fix

// before: policy lacks delete
{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":"arn:aws:s3:::bucket/*"}
// after
{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":"arn:aws:s3:::bucket/*"}

use aws_sdk_s3::Client;

async fn delete_permission_ok(client: &Client, bucket: &str) -> anyhow::Result<()> {
    // canary single-key delete surfaces IAM problems before a large batch runs
    client.delete_object().bucket(bucket).key("__perm_probe__").send().await?;
    Ok(())
}

fn is_partial_delete_failure(err: &anyhow::Error) -> bool {
    err.to_string().starts_with("Failed to delete")
}

let mut attempt = 0;
loop {
    match storage.delete_objects(&paths, &cancel).await {
        Ok(()) => break,
        Err(e) if is_partial_delete_failure(&e) && attempt < 3 => {
            attempt += 1;
            tokio::time::sleep(std::time::Duration::from_secs(1 << attempt)).await;
        }
        Err(e) => return Err(e.context("delete_objects failed permanently")),
    }
}

When it happens

Trigger: Calling delete/delete_objects (e.g. pageserver GC deleting layers) when the IAM role lacks s3:DeleteObject, objects are under object-lock/legal hold, the bucket policy forbids delete, or S3 returns transient InternalError for a subset of keys.

Common situations: GC starts failing after an IAM policy change; object lock enabled on the bucket; S3-compatible providers returning partial failures in multi-delete requests.

Related errors

• Missing ETag header
• Missing LastModified header
• Received truncated ListObjectVersions response for prefix={p
• head_object doesn't contain last_modified or content_length
• Read back file doesn't match original

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/113e41104d793bd2. Report an issue: GitHub.