apache/druid · error · SegmentLoadingException

Unable to kill segment

Error message

Unable to kill segment

What it means

HdfsDataSegmentKiller.kill() wraps any IOException raised while deleting a segment file, its descriptor.json, or its empty parent directories on HDFS into a SegmentLoadingException with the message "Unable to kill segment". The original IOException is carried as the cause, so the real problem (connection loss, permission denial, lease/recovery issues) is only visible in the cause chain. This is a generic wrapper around HDFS delete operations, distinct from the more specific 'failed to delete [%s]' message at HdfsDataSegmentKiller.java:113.

Source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java:125

                  segmentParts[0],
                  segmentParts.length == 2 ? "" : segmentParts[1] + "_"
              )
          );
        }

        if (!fs.delete(segmentPath, false)) {
          throw new SegmentLoadingException("Unable to kill segment, failed to delete [%s]", segmentPath.toString());
        }

        // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored
        // anymore, but we still delete them if exists.
        fs.delete(descriptorPath, false);

        removeEmptyParentDirectories(fs, segmentPath, segmentParts.length > 1 ? 2 : 3);
      }
    }
    catch (IOException e) {
      throw new SegmentLoadingException(e, "Unable to kill segment");
    }
  }

  @Override
  public void killAll() throws IOException
  {
    if (storageDirectory == null) {
      throw new ISE("Cannot delete all segment files since druid.storage.storageDirectory is not set.");
    }

    log.info("Deleting all segment files from hdfs dir [%s].", storageDirectory.toUri().toString());
    final FileSystem fs = storageDirectory.getFileSystem(config);
    fs.delete(storageDirectory, true);
  }

  private void removeEmptyParentDirectories(final FileSystem fs, final Path segmentPath, final int depth)
  {
    Path path = segmentPath;

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect the cause chain (SegmentLoadingException.getCause()) for the underlying IOException and fix that root problem first (connectivity, permissions, safe mode).
2. Verify the Druid process has HDFS write/delete permission on the segment path and its parent directories (check dfs.permissions and the submitting user's proxy groups).
3. Check NameNode health and exit safe mode if active (`hdfs dfsadmin -safemode leave`), and confirm the target directory is not inside a snapshot.
4. Retry the kill operation — many causes are transient HDFS RPC/network failures; Druid kill tasks can simply be re-run.
5. Check core-site.xml/hdfs-site.xml are on the classpath of the node running the kill so the correct FileSystem is resolved.

Example fix

// before: kill fails with opaque message while cluster in safe mode
// $ hdfs dfsadmin -report  ->  Safe mode is ON
// after
// $ sudo -u hdfs hdfs dfsadmin -safemode leave
// then re-run the kill task / segment deletion

Path p = new Path(String.valueOf(segment.getLoadSpec().get("path")));
try (FileSystem fs = p.getFileSystem(config)) {
  if (!fs.exists(p)) { return; } // nothing to kill
}

if (segment.getLoadSpec() == null || segment.getLoadSpec().get("path") == null) { skip / log and return; }

try { killer.kill(segment); }
catch (SegmentLoadingException e) {
  log.error(e.getCause(), "Kill failed for %s", segment.getId());
  if (isTransient(e.getCause())) { retryLater(segment); }
}

When it happens

Trigger: Calling kill(DataSegment) when fs.delete(segmentPath), fs.delete(descriptorPath), or removeEmptyParentDirectories' fs.listStatus/fs.delete throws IOException — e.g. NameNode unreachable, HDFS permission denied, snapshot on the path, or a filesystem RPC failure during cleanup of empty parent dirs.

Common situations: Coordinator issuing kill tasks while the HDFS cluster is degraded or in safe mode; HDFS permissions changed after segments were pushed; the segment directory is inside an HDFS snapshot (deletion forbidden); transient network partitions between Druid historical/coordinator nodes and the NameNode.

Related errors

• Couldn't delete segments from Azure. See the task logs for m
• No files found at [%s]
• Error loading [%s]
• Don't know how to load SCHEME [%s] for URI [%s]
• Segment path [%s] does not exist

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/c7099304cba3040c. Report an issue: GitHub.