What causes "Failed to delete {}/{} objects" in neondatabase/neon?

Read the preceding warn logs — they contain the exact failing keys and GCS error codes

Failed to delete {}/{} objects

Error message

Failed to delete {}/{} objects

What it means

The GCS batch delete response reported one or more per-object failures (the JSON API batch endpoint returns per-key results with error codes). The code logs up to 10 failing keys with their codes as warnings, then fails the whole chunk with 'Failed to delete {failed}/{total} objects'.

Source

Thrown at libs/remote_storage/src/gcs_bucket.rs:583

                    } else {
                        None
                    }
                })
                .collect();

            if !errors.is_empty() {
                // Report 10 of them like S3
                const LOG_UP_TO_N_ERRORS: usize = 10;
                for (id, code) in errors.iter().take(LOG_UP_TO_N_ERRORS) {
                    tracing::warn!(
                        "DeleteObjects key {} failed with code: {}",
                        delete_objects_status.get(id).unwrap(),
                        code
                    );
                }

                return Err(anyhow::anyhow!(
                    "Failed to delete {}/{} objects",
                    errors.len(),
                    chunk.len(),
                ));
            }
        }

        Ok(())
    }

    async fn head_object(
        &self,
        key: String,
        cancel: &CancellationToken,
    ) -> Result<GCSObject, DownloadError> {
        let kind = RequestKind::Head;
        let _permit = self.permit(kind, cancel).await?;

        let encoded_path: String = url::form_urlencoded::byte_serialize(key.as_bytes()).collect();

View on GitHub (pinned to 8f60b04da4)

Solutions

Read the preceding warn logs — they contain the exact failing keys and GCS error codes
Verify the identity has storage.objects.delete on the bucket
Retry the failed subset with backoff — deleting an already-deleted object is a success in GCS, so retries converge
If racing with writers, re-run the deletion later as a sweep rather than failing the whole operation

Defensive patterns

Strategy: retry

Try / catch

// Partial batch failure: inspect the warned codes, retry the chunk; already-deleted keys converge.
async fn delete_with_retry(storage: &Arc<GenericRemoteStorage>, oids: &[String], cancel: &CancellationToken) -> anyhow::Result<()> {
    let mut pending: Vec<String> = oids.to_vec();
    for attempt in 0..3 {
        if pending.is_empty() { return Ok(()); }
        match storage.delete_objects(&pending, cancel).await {
            Ok(()) => return Ok(()),
            Err(e) if format!("{e:#}").contains("Failed to delete") && attempt < 2 => {
                tracing::warn!("batch delete partial failure (attempt {attempt}), retrying {} keys", pending.len());
                tokio::time::sleep(Duration::from_secs(1)).await;
            }
            Err(e) => return Err(e),
        }
    }
    unreachable!()
}

Prevention

Grant storage.objects.delete to every identity that runs GC/deletion
Watch the per-key warn logs — they identify exactly which objects and codes failed
Design GC sweeps to tolerate and re-sweep partial failures instead of failing the whole pass

When it happens

Trigger: delete_objects where any object in the batch returns an error code — service account missing storage.objects.delete, precondition failures, or transient per-key errors while other keys in the same batch succeeded.

Common situations: IAM role lacking delete permission (all keys fail); GC racing with concurrent writers/modifiers (some keys fail); partial network failures mid-batch.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/670d14b8e1fb66be. Report an issue: GitHub.