apache/druid · error · SegmentLoadingException

Couldn't delete segments from S3. See the task logs for more

Error message

Couldn't delete segments from S3. See the task logs for more details.

What it means

S3DataSegmentKiller.killRetry aggregates failures from attempting to delete each file of a segment; if any deletion failed it throws this generic SegmentLoadingException instead of the detailed cause. The real reason is only in the task logs, as the code comment notes.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java:133

        keysToDelete.add(ObjectIdentifier.builder().key(path).build());
        keysToDelete.add(ObjectIdentifier.builder().key(DataSegmentKiller.descriptorPath(path)).build());
      }
    }

    boolean shouldThrowException = false;
    for (Map.Entry<String, List<ObjectIdentifier>> bucketToKeys : bucketToKeysToDelete.entrySet()) {
      String s3Bucket = bucketToKeys.getKey();
      List<ObjectIdentifier> keysToDelete = bucketToKeys.getValue();
      boolean hadException = deleteKeysForBucket(s3Client, s3Bucket, keysToDelete);
      if (hadException) {
        shouldThrowException = true;
      }
    }
    if (shouldThrowException) {
      // exception error message gets cutoff without providing any details. look at the logs for more details.
      // this was a shortcut to handle the many different ways there could potentially be failures and handle them
      // reasonably
      throw new SegmentLoadingException(
          "Couldn't delete segments from S3. See the task logs for more details."
      );
    }
  }

  /**
   * Delete all keys in a bucket from s3
   *
   * @param s3Client     client used to communicate with s3
   * @param s3Bucket     the bucket where the keys exist
   * @param keysToDelete the keys to delete
   * @return a boolean value of true if there was an issue deleting one or many keys, a boolean value of false if
   * succesful
   */
  private boolean deleteKeysForBucket(
      ServerSideEncryptingAmazonS3 s3Client,
      String s3Bucket,
      List<ObjectIdentifier> keysToDelete

View on GitHub (pinned to 9b90983fd2)

Solutions

Read the task logs: the underlying exception per file is logged there.
Grant the service account s3:DeleteObject on the segment prefix.
Check if another process already deleted the segments (race is often benign).
Retry the kill; throttling-related failures succeed on retry.
Verify druid.storage.bucket/baseKey point to the same location the segments were pushed to.

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check delete permission
s3Client.headObject(b, key); // ensures readable; delete perms must be verified via policy review

Try / catch

try { killer.kill(segment); } catch (SegmentLoadingException e) { inspectTaskLogs(taskId); }

Prevention

Grant s3:DeleteObject on the deep storage prefix
Avoid concurrent kill tasks for the same segments
Always read task logs — this message intentionally hides details

When it happens

Trigger: kill() iterates the segment's files (and descriptor) deleting each from S3; any per-file delete throwing an exception sets shouldThrowException=true and this error is thrown after all attempts.

Common situations: Missing s3:DeleteObject permission, objects deleted concurrently by another killer, S3 throttling during mass kill, wrong bucket/baseKey config.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/b1b4dcfe76f77f45. Report an issue: GitHub.