{"record":{"id":"1c72e4403cc9213a","repo":"apache/iceberg","slug":"caught-unexpected-exception-during-batch-deletion","errorCode":null,"errorMessage":"Caught unexpected exception during batch deletion: ","messagePattern":"Caught unexpected exception during batch deletion: ","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java","lineNumber":260,"sourceCode":"      for (Map.Entry<String, Collection<String>> bucketToObjectsEntry :\n          bucketToObjects.asMap().entrySet()) {\n        String bucket = bucketToObjectsEntry.getKey();\n        Collection<String> keys = bucketToObjectsEntry.getValue();\n        Future<List<String>> deletionTask =\n            executorService()\n                .submit(() -> deleteBatch(clientForStoragePath(\"s3://\" + bucket), bucket, keys));\n        deletionTasks.add(deletionTask);\n      }\n\n      int totalFailedDeletions = 0;\n\n      for (Future<List<String>> deletionTask : deletionTasks) {\n        try {\n          List<String> failedDeletions = deletionTask.get();\n          failedDeletions.forEach(path -> LOG.warn(\"Failed to delete object at path {}\", path));\n          totalFailedDeletions += failedDeletions.size();\n        } catch (ExecutionException e) {\n          LOG.warn(\"Caught unexpected exception during batch deletion: \", e.getCause());\n        } catch (InterruptedException e) {\n          Thread.currentThread().interrupt();\n          deletionTasks.stream().filter(task -> !task.isDone()).forEach(task -> task.cancel(true));\n          throw new RuntimeException(\"Interrupted when waiting for deletions to complete\", e);\n        }\n      }\n\n      if (totalFailedDeletions > 0) {\n        throw new BulkDeletionFailureException(totalFailedDeletions);\n      }\n    }\n  }\n\n  private void tagFileToDelete(PrefixedS3Client client, String path, Set<Tag> deleteTags)\n      throws S3Exception {\n    S3URI location = new S3URI(path, client.s3FileIOProperties().bucketToAccessPointMapping());\n    String bucket = location.bucket();\n    String objectKey = location.key();","sourceCodeStart":242,"sourceCodeEnd":278,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java#L242-L278","documentation":"S3FileIO.deleteFiles submits batch deletion tasks and, when collecting results, a task failing with an ExecutionException is not rethrown — the cause is logged at WARN with 'Caught unexpected exception during batch deletion: ' and counted implicitly via failed deletions. This means objects may remain in S3 after deleteFiles returns; the error is surfaced only in logs, so silent data retention is possible.","triggerScenarios":"Concurrent S3 batch deletion (DeleteObjects calls via Tasks.groupby) where a worker task throws — e.g. S3 access denied, throttling (SlowDown), missing object permissions, or an SDK client failure — inside one of the deletionTasks futures.","commonSituations":"Deleting large manifest batches with credentials lacking s3:DeleteObject on some keys; S3 rate limiting during mass deletes; transient network/SDK errors mid-batch; expired session credentials partway through a long cleanup.","solutions":["Inspect application logs for the WARN entry and its cause to identify failed object keys","Re-run deleteFiles after fixing the underlying cause (permissions, throttling) — deletion is idempotent for already-deleted objects","Increase batch deletion parallelism/limits or add retry/backoff for SlowDown errors","Grant s3:DeleteObject on the full prefix in the bucket policy / IAM role"],"exampleFix":"// before: assuming all deletes succeeded\nfileIO.deleteFiles(tasks);\n// after: verify via logs or re-check\nfileIO.deleteFiles(tasks);\nLOG.info(\"Batch deletion finished; check logs for 'Caught unexpected exception during batch deletion' and re-run for leftovers\");","handlingStrategy":"retry","validationCode":"// pre-check permissions/scope before mass deletion\ndeleteCandidates.forEach(key ->\n  require(key.startsWith(\"s3://\" + bucket + \"/\" + allowedPrefix), \"key outside allowed prefix: \" + key));","typeGuard":null,"tryCatchPattern":"List<String> leftovers = fileIO.deleteFiles(tasks);\nif (!leftovers.isEmpty()) {\n  // re-attempt idempotent deletion after fixing cause (IAM, throttling)\n  leftovers.forEach(path -> LOG.error(\"object not deleted: {}\", path));\n}\n// and monitor logs for \"Caught unexpected exception during batch deletion\"","preventionTips":["Grant s3:DeleteObject on the entire prefix used by the table","Tune batch size/parallelism to avoid S3 SlowDown throttling","Re-run deleteFiles after failures — object deletion is idempotent","Alert on the WARN log message during bulk cleanup jobs"],"tags":["aws","s3","batch-deletion","concurrency"],"backgroundTag":"s3-batch-delete-failure","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}