{"record":{"id":"ea8720f4d0645f43","repo":"apache/druid","slug":"couldn-t-kill-segment-s-s-ea8720","errorCode":null,"errorMessage":"Couldn't kill segment[%s]: [%s]","messagePattern":"Couldn't kill segment\\[(.+?)\\]: \\[(.+?)\\]","errorType":"exception","errorClass":"SegmentLoadingException","httpStatus":null,"severity":"error","filePath":"extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java","lineNumber":208,"sourceCode":"          log.info(\"Removing index file[s3://%s/%s] from s3!\", s3Bucket, objectSummary.key());\n          s3Client.deleteObject(s3Bucket, objectSummary.key());\n        }\n      } else {\n        String s3DescriptorPath = DataSegmentKiller.descriptorPath(s3Path);\n        if (s3Client.doesObjectExist(s3Bucket, s3Path)) {\n          log.info(\"Removing index file[s3://%s/%s] from s3!\", s3Bucket, s3Path);\n          s3Client.deleteObject(s3Bucket, s3Path);\n        }\n        // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored\n        // anymore, but we still delete them if exists.\n        if (s3Client.doesObjectExist(s3Bucket, s3DescriptorPath)) {\n          log.info(\"Removing descriptor file[s3://%s/%s] from s3!\", s3Bucket, s3DescriptorPath);\n          s3Client.deleteObject(s3Bucket, s3DescriptorPath);\n        }\n      }\n    }\n    catch (S3Exception e) {\n      throw new SegmentLoadingException(e, \"Couldn't kill segment[%s]: [%s]\", segment.getId(), e);\n    }\n  }\n\n  @Override\n  public void killAll() throws IOException\n  {\n    if (segmentPusherConfig.getBucket() == null || segmentPusherConfig.getBaseKey() == null) {\n      throw new ISE(\n          \"Cannot delete all segment from S3 Deep Storage since druid.storage.bucket and druid.storage.baseKey are not both set.\");\n    }\n    log.info(\"Deleting all segment files from s3 location [bucket: '%s' prefix: '%s']\",\n             segmentPusherConfig.getBucket(), segmentPusherConfig.getBaseKey()\n    );\n    try {\n      S3Utils.deleteObjectsInPath(\n          s3ClientSupplier.get(),\n          inputDataConfig.getMaxListingLength(),\n          segmentPusherConfig.getBucket(),","sourceCodeStart":190,"sourceCodeEnd":226,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java#L190-L226","documentation":"Thrown when the AWS S3 client raises S3Exception while deleting the files (and optionally descriptor) of one segment during killRetry. It wraps the SDK exception with the segment id and the exception text.","triggerScenarios":"kill() calls s3Client.deleteObject (copy then delete for descriptor) and the SDK throws S3Exception: access denied, key not found in strict mode, throttling, or invalid encryption settings.","commonSituations":"IAM policy lacking s3:DeleteObject, segments already removed by concurrent kill tasks, S3 503 slow-down during bulk operations, SSE-KMS key policy changes.","solutions":["Inspect the wrapped S3Exception message/status in the error for the precise cause.","Add s3:DeleteObject (and s3:PutObject for the descriptor copy) to the IAM policy.","If 503/SlowDown, reduce kill concurrency or retry later.","Confirm the segment's loadSpec bucket/key match actual S3 layout."],"exampleFix":"// IAM policy addition\n// before: read-only policy\n// after:\n// \"Statement\": [{ \"Effect\": \"Allow\", \"Action\": [\"s3:DeleteObject\", \"s3:PutObject\"], \"Resource\": \"arn:aws:s3:::druid-bucket/segments/*\" }]","handlingStrategy":"try-catch","validationCode":"// verify object exists and is deletable via IAM simulation\ntry { s3Client.headObject(bucket, key); } catch (NoSuchKeyException e) { /* already gone, skip */ }","typeGuard":null,"tryCatchPattern":"try { killer.kill(segment); } catch (SegmentLoadingException e) { if (isThrottling(e.getCause())) backoffRetry(); else alert(e.getCause()); }","preventionTips":["Include s3:DeleteObject and s3:PutObject in the killer task's IAM role","Keep KMS key policies consistent if SSE-KMS is used","Throttle bulk kill operations to avoid 503 SlowDown"],"tags":["aws","s3","segment-kill","s3exception"],"backgroundTag":"http-error-response","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}