{"record":{"id":"a461ead8315c15a0","repo":"apache/iceberg","slug":"bulk-deletion-failed-for-of-file-s","errorCode":null,"errorMessage":"Bulk deletion failed for {} of {} {} file(s)","messagePattern":"Bulk deletion failed for (.+?) of (.+?) (.+?) file\\(s\\)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"core/src/main/java/org/apache/iceberg/FileCleanupStrategy.java","lineNumber":112,"sourceCode":"  protected CloseableIterable<ManifestFile> readManifests(Snapshot snapshot) {\n    if (snapshot.manifestListLocation() != null) {\n      return InternalData.read(\n              FileFormat.AVRO, fileIO.newInputFile(snapshot.manifestListLocation()))\n          .setRootType(GenericManifestFile.class)\n          .project(MANIFEST_PROJECTION)\n          .reuseContainers()\n          .build();\n    } else {\n      return CloseableIterable.withNoopClose(snapshot.allManifests(fileIO));\n    }\n  }\n\n  protected void deleteFiles(Set<String> pathsToDelete, String fileType) {\n    if (deleteFunc == null && fileIO instanceof SupportsBulkOperations) {\n      try {\n        ((SupportsBulkOperations) fileIO).deleteFiles(pathsToDelete);\n      } catch (BulkDeletionFailureException e) {\n        LOG.warn(\n            \"Bulk deletion failed for {} of {} {} file(s)\",\n            e.numberFailedObjects(),\n            pathsToDelete.size(),\n            fileType,\n            e);\n      } catch (RuntimeException e) {\n        LOG.warn(\"Bulk deletion failed\", e);\n      }\n    } else {\n      Consumer<String> deleteFuncToUse = deleteFunc == null ? defaultDeleteFunc : deleteFunc;\n\n      Tasks.foreach(pathsToDelete)\n          .executeWith(deleteExecutorService)\n          .retry(3)\n          .stopRetryOn(NotFoundException.class)\n          .stopOnFailure()\n          .suppressFailureWhenFinished()\n          .onFailure(","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/FileCleanupStrategy.java#L94-L130","documentation":"A WARN from FileCleanupStrategy.deleteFiles (used by expireSnapshots and orphan/planning cleanup jobs) when bulk deletion via SupportsBulkOperations reports BulkDeletionFailureException: a subset of the pathsToDelete could not be deleted, with counts given in the message. Cleanup proceeds as best-effort; the failed files remain on storage until a future cleanup run deletes them.","triggerScenarios":"expireSnapshots (or similar cleanup) with a bulk-capable FileIO; the object store's bulk delete fails for some objects — throttling, permissions on individual keys, objects concurrently deleted by another cleanup job, or batch failures in the S3/GCS client.","commonSituations":"Snapshot expiration under aggressive S3 rate limits; overlapping expireSnapshots jobs from different clients; bucket policies denying deletes for certain prefixes; large batches exceeding provider batch limits.","solutions":["Re-run expireSnapshots/cleanup — already-deleted files are skipped and remaining ones retried.","Throttle cleanup concurrency or run during off-peak hours to avoid provider rate limiting.","Ensure only one cleanup job runs per table at a time to avoid racing deletions.","Review IAM policies for delete permission on the data and metadata prefixes."],"exampleFix":"// before: overlapping expire jobs\nspark.submit(expireJob); anotherClient.expireSnapshots(table);\n// after: serialize cleanup runs\nif (lockAcquired(table)) {\n  SparkActions.get().expireSnapshots(table).olderThan(ts).execute();\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { actions.expireSnapshots(table).olderThan(ts).execute(); } catch (RuntimeException e) { /* re-run; deletes are idempotent */ }","preventionTips":["Run a single expireSnapshots job per table at a time","Throttle/rate-limit cleanup to avoid object-store throttling","Re-run expiration after partial bulk-delete failures","Verify delete permissions on data and metadata prefixes"],"tags":["bulk-delete","snapshot-expiration","cleanup"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}