{"record":{"id":"b1b4dcfe76f77f45","repo":"apache/druid","slug":"couldn-t-delete-segments-from-s3-see-the-task-log","errorCode":null,"errorMessage":"Couldn't delete segments from S3. See the task logs for more details.","messagePattern":"Couldn't delete segments from S3\\. See the task logs for more details\\.","errorType":"exception","errorClass":"SegmentLoadingException","httpStatus":null,"severity":"error","filePath":"extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java","lineNumber":133,"sourceCode":"        keysToDelete.add(ObjectIdentifier.builder().key(path).build());\n        keysToDelete.add(ObjectIdentifier.builder().key(DataSegmentKiller.descriptorPath(path)).build());\n      }\n    }\n\n    boolean shouldThrowException = false;\n    for (Map.Entry<String, List<ObjectIdentifier>> bucketToKeys : bucketToKeysToDelete.entrySet()) {\n      String s3Bucket = bucketToKeys.getKey();\n      List<ObjectIdentifier> keysToDelete = bucketToKeys.getValue();\n      boolean hadException = deleteKeysForBucket(s3Client, s3Bucket, keysToDelete);\n      if (hadException) {\n        shouldThrowException = true;\n      }\n    }\n    if (shouldThrowException) {\n      // exception error message gets cutoff without providing any details. look at the logs for more details.\n      // this was a shortcut to handle the many different ways there could potentially be failures and handle them\n      // reasonably\n      throw new SegmentLoadingException(\n          \"Couldn't delete segments from S3. See the task logs for more details.\"\n      );\n    }\n  }\n\n  /**\n   * Delete all keys in a bucket from s3\n   *\n   * @param s3Client     client used to communicate with s3\n   * @param s3Bucket     the bucket where the keys exist\n   * @param keysToDelete the keys to delete\n   * @return a boolean value of true if there was an issue deleting one or many keys, a boolean value of false if\n   * succesful\n   */\n  private boolean deleteKeysForBucket(\n      ServerSideEncryptingAmazonS3 s3Client,\n      String s3Bucket,\n      List<ObjectIdentifier> keysToDelete","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java#L115-L151","documentation":"S3DataSegmentKiller.killRetry aggregates failures from attempting to delete each file of a segment; if any deletion failed it throws this generic SegmentLoadingException instead of the detailed cause. The real reason is only in the task logs, as the code comment notes.","triggerScenarios":"kill() iterates the segment's files (and descriptor) deleting each from S3; any per-file delete throwing an exception sets shouldThrowException=true and this error is thrown after all attempts.","commonSituations":"Missing s3:DeleteObject permission, objects deleted concurrently by another killer, S3 throttling during mass kill, wrong bucket/baseKey config.","solutions":["Read the task logs: the underlying exception per file is logged there.","Grant the service account s3:DeleteObject on the segment prefix.","Check if another process already deleted the segments (race is often benign).","Retry the kill; throttling-related failures succeed on retry.","Verify druid.storage.bucket/baseKey point to the same location the segments were pushed to."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-check delete permission\ns3Client.headObject(b, key); // ensures readable; delete perms must be verified via policy review","typeGuard":null,"tryCatchPattern":"try { killer.kill(segment); } catch (SegmentLoadingException e) { inspectTaskLogs(taskId); }","preventionTips":["Grant s3:DeleteObject on the deep storage prefix","Avoid concurrent kill tasks for the same segments","Always read task logs — this message intentionally hides details"],"tags":["aws","s3","segment-kill","cleanup"],"backgroundTag":"http-error-response","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}