apache/druid · error · SegmentLoadingException

Couldn't kill segment[%s]: [%s]

Error message

Couldn't kill segment[%s]: [%s]

What it means

Thrown when the AWS S3 client raises S3Exception while deleting the files (and optionally descriptor) of one segment during killRetry. It wraps the SDK exception with the segment id and the exception text.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java:208

          log.info("Removing index file[s3://%s/%s] from s3!", s3Bucket, objectSummary.key());
          s3Client.deleteObject(s3Bucket, objectSummary.key());
        }
      } else {
        String s3DescriptorPath = DataSegmentKiller.descriptorPath(s3Path);
        if (s3Client.doesObjectExist(s3Bucket, s3Path)) {
          log.info("Removing index file[s3://%s/%s] from s3!", s3Bucket, s3Path);
          s3Client.deleteObject(s3Bucket, s3Path);
        }
        // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored
        // anymore, but we still delete them if exists.
        if (s3Client.doesObjectExist(s3Bucket, s3DescriptorPath)) {
          log.info("Removing descriptor file[s3://%s/%s] from s3!", s3Bucket, s3DescriptorPath);
          s3Client.deleteObject(s3Bucket, s3DescriptorPath);
        }
      }
    }
    catch (S3Exception e) {
      throw new SegmentLoadingException(e, "Couldn't kill segment[%s]: [%s]", segment.getId(), e);
    }
  }

  @Override
  public void killAll() throws IOException
  {
    if (segmentPusherConfig.getBucket() == null || segmentPusherConfig.getBaseKey() == null) {
      throw new ISE(
          "Cannot delete all segment from S3 Deep Storage since druid.storage.bucket and druid.storage.baseKey are not both set.");
    }
    log.info("Deleting all segment files from s3 location [bucket: '%s' prefix: '%s']",
             segmentPusherConfig.getBucket(), segmentPusherConfig.getBaseKey()
    );
    try {
      S3Utils.deleteObjectsInPath(
          s3ClientSupplier.get(),
          inputDataConfig.getMaxListingLength(),
          segmentPusherConfig.getBucket(),

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect the wrapped S3Exception message/status in the error for the precise cause.
2. Add s3:DeleteObject (and s3:PutObject for the descriptor copy) to the IAM policy.
3. If 503/SlowDown, reduce kill concurrency or retry later.
4. Confirm the segment's loadSpec bucket/key match actual S3 layout.

Example fix

// IAM policy addition
// before: read-only policy
// after:
// "Statement": [{ "Effect": "Allow", "Action": ["s3:DeleteObject", "s3:PutObject"], "Resource": "arn:aws:s3:::druid-bucket/segments/*" }]

// verify object exists and is deletable via IAM simulation
try { s3Client.headObject(bucket, key); } catch (NoSuchKeyException e) { /* already gone, skip */ }

try { killer.kill(segment); } catch (SegmentLoadingException e) { if (isThrottling(e.getCause())) backoffRetry(); else alert(e.getCause()); }

When it happens

Trigger: kill() calls s3Client.deleteObject (copy then delete for descriptor) and the SDK throws S3Exception: access denied, key not found in strict mode, throttling, or invalid encryption settings.

Common situations: IAM policy lacking s3:DeleteObject, segments already removed by concurrent kill tasks, S3 503 slow-down during bulk operations, SSE-KMS key policy changes.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Couldn't delete segments from S3. See the task logs for more
• Unable to move segment[%s]: [%s]
• Failed to get object summaries from S3 bucket[%s], prefix[%s
• Could not fetch last modified timestamp from URI [%s]
• Error fetching last modified timestamp from URI [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ea8720f4d0645f43. Report an issue: GitHub.