apache/druid · error · SegmentLoadingException

Unable to kill segment, failed to delete [%s]

Error message

Unable to kill segment, failed to delete [%s]

What it means

After removing descriptor.json paths, HdfsDataSegmentKiller.kill calls fs.delete(segmentPath, false) and throws this SegmentLoadingException if HDFS reports the deletion did not succeed. This usually means the file is missing, protected by permissions, or the NameNode is in an unhealthy state — the segment was not killed.

Source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java:114

                                   StringUtils.isNumeric(segmentParts[0]) &&
                                   ("index" + compressionFormat.getSuffix()).equals(
                                       segmentParts[segmentParts.length - 1]
                                   ),
                                   "Unexpected segmentPath format [%s]", segmentPath
          );

          descriptorPath = new Path(
              segmentPath.getParent(),
              org.apache.druid.java.util.common.StringUtils.format(
                  "%s_%sdescriptor.json",
                  segmentParts[0],
                  segmentParts.length == 2 ? "" : segmentParts[1] + "_"
              )
          );
        }

        if (!fs.delete(segmentPath, false)) {
          throw new SegmentLoadingException("Unable to kill segment, failed to delete [%s]", segmentPath.toString());
        }

        // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored
        // anymore, but we still delete them if exists.
        fs.delete(descriptorPath, false);

        removeEmptyParentDirectories(fs, segmentPath, segmentParts.length > 1 ? 2 : 3);
      }
    }
    catch (IOException e) {
      throw new SegmentLoadingException(e, "Unable to kill segment");
    }
  }

  @Override
  public void killAll() throws IOException
  {
    if (storageDirectory == null) {

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Check the file exists first: hdfs dfs -ls <segmentPath>; if already gone, refresh metadata store/coordinator instead of killing.
2. Check permissions: the Druid process user needs write permission on the file's parent directory (hdfs dfs -chmod/chown as needed).
3. Look for HDFS-level causes — safe mode, snapshots, trash, protection policies (Ranger/Sentry) — in the NameNode logs.
4. Retry after the cluster is healthy if NameNode errors were reported.
5. Verify deletion afterwards (hdfs dfs -test -e) and alert if the path persists.

Example fix

// before
if (!fs.delete(segmentPath, false)) {
  throw new SegmentLoadingException("Unable to kill segment, failed to delete [%s]", segmentPath.toString());
}
// after
if (fs.exists(segmentPath)) {
  if (!fs.delete(segmentPath, false)) {
    throw new SegmentLoadingException("Unable to kill segment, failed to delete [%s]", segmentPath);
  }
} else {
  log.warn("Segment [%s] already gone", segmentPath);
}

FileSystem fs = segmentPath.getFileSystem(config);
if (!fs.exists(segmentPath)) { /* already gone; refresh metadata instead of kill */ }
else if (!fs.getParent(segmentPath).getFileSystem(config)
            .getFileStatus(fs.getParent(segmentPath)).getPermission().applyUMask(null).getUserAction().implies(FsAction.WRITE)) {
  /* fix permissions first */ }

try { killer.kill(segment); }
catch (SegmentLoadingException e) {
  retryWithBackoff(() -> killer.kill(segment)); // or fix permissions/refresh metadata
}

When it happens

Trigger: Calling kill() when the segment file does not actually exist at the path, the Druid user lacks write permission on the parent directory, the file is in a snapshot/protected directory, or HDFS returns false due to NameNode errors.

Common situations: Stale segment metadata pointing to already-deleted files; deep-storage directories owned by another user after a migration; HDFS permissions/sentry/ranger policies blocking deletion; NameNode safe-mode or decommissioned datanodes during delete.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Unknown file type[%s]
• Failed to delete deep storage directory[<dirToDelete>].
• S3 multi-object delete had <n> error(s):\n key=[<key>], cod
• Segment path [%s] does not exist
• Skipping deep storage directory kill: relative path is empty

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/20de95de50c0238f. Report an issue: GitHub.