{"record":{"id":"c7099304cba3040c","repo":"apache/druid","slug":"unable-to-kill-segment","errorCode":null,"errorMessage":"Unable to kill segment","messagePattern":"Unable to kill segment","errorType":"exception","errorClass":"SegmentLoadingException","httpStatus":null,"severity":"error","filePath":"extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java","lineNumber":125,"sourceCode":"                  segmentParts[0],\n                  segmentParts.length == 2 ? \"\" : segmentParts[1] + \"_\"\n              )\n          );\n        }\n\n        if (!fs.delete(segmentPath, false)) {\n          throw new SegmentLoadingException(\"Unable to kill segment, failed to delete [%s]\", segmentPath.toString());\n        }\n\n        // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored\n        // anymore, but we still delete them if exists.\n        fs.delete(descriptorPath, false);\n\n        removeEmptyParentDirectories(fs, segmentPath, segmentParts.length > 1 ? 2 : 3);\n      }\n    }\n    catch (IOException e) {\n      throw new SegmentLoadingException(e, \"Unable to kill segment\");\n    }\n  }\n\n  @Override\n  public void killAll() throws IOException\n  {\n    if (storageDirectory == null) {\n      throw new ISE(\"Cannot delete all segment files since druid.storage.storageDirectory is not set.\");\n    }\n\n    log.info(\"Deleting all segment files from hdfs dir [%s].\", storageDirectory.toUri().toString());\n    final FileSystem fs = storageDirectory.getFileSystem(config);\n    fs.delete(storageDirectory, true);\n  }\n\n  private void removeEmptyParentDirectories(final FileSystem fs, final Path segmentPath, final int depth)\n  {\n    Path path = segmentPath;","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java#L107-L143","documentation":"HdfsDataSegmentKiller.kill() wraps any IOException raised while deleting a segment file, its descriptor.json, or its empty parent directories on HDFS into a SegmentLoadingException with the message \"Unable to kill segment\". The original IOException is carried as the cause, so the real problem (connection loss, permission denial, lease/recovery issues) is only visible in the cause chain. This is a generic wrapper around HDFS delete operations, distinct from the more specific 'failed to delete [%s]' message at HdfsDataSegmentKiller.java:113.","triggerScenarios":"Calling kill(DataSegment) when fs.delete(segmentPath), fs.delete(descriptorPath), or removeEmptyParentDirectories' fs.listStatus/fs.delete throws IOException — e.g. NameNode unreachable, HDFS permission denied, snapshot on the path, or a filesystem RPC failure during cleanup of empty parent dirs.","commonSituations":"Coordinator issuing kill tasks while the HDFS cluster is degraded or in safe mode; HDFS permissions changed after segments were pushed; the segment directory is inside an HDFS snapshot (deletion forbidden); transient network partitions between Druid historical/coordinator nodes and the NameNode.","solutions":["Inspect the cause chain (SegmentLoadingException.getCause()) for the underlying IOException and fix that root problem first (connectivity, permissions, safe mode).","Verify the Druid process has HDFS write/delete permission on the segment path and its parent directories (check dfs.permissions and the submitting user's proxy groups).","Check NameNode health and exit safe mode if active (`hdfs dfsadmin -safemode leave`), and confirm the target directory is not inside a snapshot.","Retry the kill operation — many causes are transient HDFS RPC/network failures; Druid kill tasks can simply be re-run.","Check core-site.xml/hdfs-site.xml are on the classpath of the node running the kill so the correct FileSystem is resolved."],"exampleFix":"// before: kill fails with opaque message while cluster in safe mode\n// $ hdfs dfsadmin -report  ->  Safe mode is ON\n// after\n// $ sudo -u hdfs hdfs dfsadmin -safemode leave\n// then re-run the kill task / segment deletion","handlingStrategy":"try-catch","validationCode":"Path p = new Path(String.valueOf(segment.getLoadSpec().get(\"path\")));\ntry (FileSystem fs = p.getFileSystem(config)) {\n  if (!fs.exists(p)) { return; } // nothing to kill\n}","typeGuard":"if (segment.getLoadSpec() == null || segment.getLoadSpec().get(\"path\") == null) { skip / log and return; }","tryCatchPattern":"try { killer.kill(segment); }\ncatch (SegmentLoadingException e) {\n  log.error(e.getCause(), \"Kill failed for %s\", segment.getId());\n  if (isTransient(e.getCause())) { retryLater(segment); }\n}","preventionTips":["Keep HDFS clients healthy and core/hdfs-site configs consistent across Druid nodes.","Grant the Druid user delete permissions on the whole deep-storage tree up front.","Avoid placing deep-storage segment trees inside HDFS snapshots.","Treat kill failures as retryable transient errors in automation."],"tags":["hdfs","segment-deletion","io-exception","deep-storage"],"backgroundTag":"file-delete-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}