{"record":{"id":"558178556d92f919","repo":"apache/iceberg","slug":"deleted-only-of-files-using-bulk-deletes-558178","errorCode":null,"errorMessage":"Deleted only {} of {} files using bulk deletes","messagePattern":"Deleted only (.+?) of (.+?) files using bulk deletes","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java","lineNumber":325,"sourceCode":"  }\n\n  private void collectPathsForOutput(\n      List<String> paths, List<String> orphanFileList, int maxSampleSize) {\n    if (streamResults()) {\n      int lengthToAdd = Math.min(maxSampleSize - orphanFileList.size(), paths.size());\n      orphanFileList.addAll(paths.subList(0, lengthToAdd));\n    } else {\n      orphanFileList.addAll(paths);\n    }\n  }\n\n  private void deleteBulk(SupportsBulkOperations io, List<String> paths) {\n    try {\n      io.deleteFiles(paths);\n      LOG.info(\"Deleted {} files using bulk deletes\", paths.size());\n    } catch (BulkDeletionFailureException e) {\n      int deletedFilesCount = paths.size() - e.numberFailedObjects();\n      LOG.warn(\n          \"Deleted only {} of {} files using bulk deletes\", deletedFilesCount, paths.size(), e);\n    }\n  }\n\n  private void deleteNonBulk(List<String> paths) {\n    Tasks.Builder<String> deleteTasks =\n        Tasks.foreach(paths)\n            .noRetry()\n            .executeWith(deleteExecutorService)\n            .suppressFailureWhenFinished()\n            .onFailure((file, exc) -> LOG.warn(\"Failed to delete file: {}\", file, exc));\n\n    if (deleteFunc == null) {\n      LOG.info(\n          \"Table IO {} does not support bulk operations. Using non-bulk deletes.\",\n          table.io().getClass().getName());\n      deleteTasks.run(table.io()::deleteFile);\n    } else {","sourceCodeStart":307,"sourceCodeEnd":343,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java#L307-L343","documentation":"A logged warning emitted by DeleteOrphanFilesSparkAction.deleteBulk when the FileIO's bulk delete (SupportsBulkOperations.deleteFiles) partially fails, signaled by BulkDeletionFailureException carrying the number of failed objects. The successfully deleted count is derived as paths.size() minus failures; remaining files are warned about but the action continues.","triggerScenarios":"Running DeleteOrphanFiles against a FileIO with bulk support (e.g., S3FileIO) where io.deleteFiles(paths) deletes some objects but fails on others — expired/short-lived credentials mid-batch, per-object permission errors, missing objects already deleted, or object-store throttling.","commonSituations":"STS credentials expiring during a long orphan-file sweep; mixed-prefix bucket policies where some keys are undeletable; concurrent DeleteOrphanFiles runs deleting the same objects; S3 503 slow-down throttling on large batches.","solutions":["Re-run deleteOrphanFiles; a second pass usually deletes the files that failed the first time.","Fix the failing keys' permissions (check the exception detail / object-store server logs for 403/404).","Reduce batch concurrency or enable S3 client retry/throttle mitigation for large sweeps.","Avoid running overlapping orphan-file deletion jobs on the same table location.","Refresh long-lived credentials or use instance profiles for multi-hour sweeps."],"exampleFix":"// before\nSparkActions.get(spark).deleteOrphanFiles().olderThan(ts).execute();  // partial bulk failure\n// after\n// inspect e.numberFailedObjects(), fix perms, then re-run the same action\nSparkActions.get(spark).deleteOrphanFiles().olderThan(ts).execute();  // retry pass","handlingStrategy":"retry","validationCode":"// confirm bulk support and credentials lifetime before sweeping\n// io instanceof SupportsBulkOperations; STS session length > expected sweep duration","typeGuard":null,"tryCatchPattern":"try { io.deleteFiles(paths); } catch (BulkDeletionFailureException e) { /* re-run for the failed objects */ }","preventionTips":["Use credentials that outlast long orphan sweeps or use instance profiles","Reduce sweep batch size to avoid object-store throttling","Re-run deleteOrphanFiles after partial failures","Avoid concurrent orphan-file deletion jobs"],"tags":["spark","actions","bulk-delete","object-store"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}