{"record":{"id":"d5f312060b77d6ef","repo":"apache/druid","slug":"s3-multi-object-delete-had-n-error-s-n-key","errorCode":null,"errorMessage":"S3 multi-object delete had <n> error(s):\\n  key=[<key>], code=[<code>], message=[<message>]","messagePattern":"S3 multi-object delete had <n> error\\(s\\):\\\\n  key=\\[<key>\\], code=\\[<code>\\], message=\\[<message>\\]","errorType":"exception","errorClass":"S3MultiObjectDeleteException","httpStatus":null,"severity":"error","filePath":"extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java","lineNumber":424,"sourceCode":"          .delete(Delete.builder().objects(remaining).build())\n          .build();\n      DeleteObjectsResponse response = retryS3Operation(() -> s3Client.deleteObjects(deleteRequest));\n      if (!response.hasErrors()) {\n        log.info(\"Deleted %d files\", keysToDelete.size());\n        return;\n      }\n      lastErrors = response.errors();\n      remaining = lastErrors.stream()\n                            .map(e -> ObjectIdentifier.builder().key(e.key()).build())\n                            .collect(Collectors.toList());\n      log.warn(\n          \"S3 multi-object delete had %d error(s) on attempt %d/%d, retrying failed keys only\",\n          remaining.size(),\n          attempt + 1,\n          retries + 1\n      );\n    }\n    throw new S3MultiObjectDeleteException(lastErrors);\n  }\n\n  /**\n   * Uploads a file to S3 if possible. First trying to set ACL to give the bucket owner full control of the file before uploading.\n   *\n   * @param service    S3 client\n   * @param disableAcl true if ACL shouldn't be set for the file\n   * @param key        The key under which to store the new object.\n   * @param file       The path of the file to upload to Amazon S3.\n   */\n  static void uploadFileIfPossible(\n      ServerSideEncryptingAmazonS3 service,\n      boolean disableAcl,\n      String bucket,\n      String key,\n      File file\n  )\n  {","sourceCodeStart":406,"sourceCodeEnd":442,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L406-L442","documentation":"S3Utils.deleteBucketKeys performs S3 multi-object delete; after exhausting retries some keys still failed, so it throws S3MultiObjectDeleteException carrying per-key error codes and messages. Only the failed keys are retried on subsequent attempts.","triggerScenarios":"Deleting objects in a path (e.g. kill task, dropping a datasource) when some keys fail DeleteObjects permanently: access denied, invalid key names, or objects protected by policies, across all retry attempts.","commonSituations":"IAM policy allowing List but not s3:DeleteObject; S3 Object Lock/versioning preventing deletion; keys with characters rejected by the API; throttling under large batch deletes.","solutions":["Inspect the exception's per-key errors (code/message) to identify failed keys.","Grant s3:DeleteObject (and DeleteObjectVersion if versioned) on the bucket/prefix.","Check for Object Lock, retention, or deny policies on the failing keys.","Re-run the delete; the failed keys are safely retried individually.","Reduce batch size if S3 throttling caused the failures."],"exampleFix":"// before\n// IAM: allow s3:ListBucket only\n// after\n// IAM: also allow\n// {\"Effect\":\"Allow\",\"Action\":[\"s3:DeleteObject\",\"s3:DeleteObjectVersion\"],\"Resource\":\"arn:aws:s3:::my-bucket/*\"}","handlingStrategy":"try-catch","validationCode":"// verify delete permission upfront\ns3Client.getBucketPolicy(b -> b.bucket(bucket)); // plus IAM dry-run where available","typeGuard":null,"tryCatchPattern":"try { S3Utils.deleteObjectsInPath(...); } catch (S3MultiObjectDeleteException e) { e.getErrors().forEach(err -> log.error(\"delete failed key=%s code=%s msg=%s\", err.key(), err.code(), err.message())); }","preventionTips":["Grant s3:DeleteObject (and versioned variant) on the prefix","Check Object Lock/retention before bulk deletes","Inspect per-key errors instead of retrying blindly"],"tags":["s3","delete","batch","permissions"],"backgroundTag":"http-error-response","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}