apache/druid · error · SegmentLoadingException

Unable to kill segment

Error message

Unable to kill segment

What it means

HdfsDataSegmentKiller.kill() wraps any IOException raised while deleting a segment file, its descriptor.json, or its empty parent directories on HDFS into a SegmentLoadingException with the message "Unable to kill segment". The original IOException is carried as the cause, so the real problem (connection loss, permission denial, lease/recovery issues) is only visible in the cause chain. This is a generic wrapper around HDFS delete operations, distinct from the more specific 'failed to delete [%s]' message at HdfsDataSegmentKiller.java:113.

Solutions

1. Inspect the cause chain (SegmentLoadingException.getCause()) for the underlying IOException and fix that root problem first (connectivity, permissions, safe mode).
2. Verify the Druid process has HDFS write/delete permission on the segment path and its parent directories (check dfs.permissions and the submitting user's proxy groups).
3. Check NameNode health and exit safe mode if active (`hdfs dfsadmin -safemode leave`), and confirm the target directory is not inside a snapshot.
4. Retry the kill operation — many causes are transient HDFS RPC/network failures; Druid kill tasks can simply be re-run.
5. Check core-site.xml/hdfs-site.xml are on the classpath of the node running the kill so the correct FileSystem is resolved.

Example fix

// before: kill fails with opaque message while cluster in safe mode
// $ hdfs dfsadmin -report  ->  Safe mode is ON
// after
// $ sudo -u hdfs hdfs dfsadmin -safemode leave
// then re-run the kill task / segment deletion

Path p = new Path(String.valueOf(segment.getLoadSpec().get("path")));
try (FileSystem fs = p.getFileSystem(config)) {
  if (!fs.exists(p)) { return; } // nothing to kill
}

if (segment.getLoadSpec() == null || segment.getLoadSpec().get("path") == null) { skip / log and return; }

try { killer.kill(segment); }
catch (SegmentLoadingException e) {
  log.error(e.getCause(), "Kill failed for %s", segment.getId());
  if (isTransient(e.getCause())) { retryLater(segment); }
}

When it happens

Trigger: Calling kill(DataSegment) when fs.delete(segmentPath), fs.delete(descriptorPath), or removeEmptyParentDirectories' fs.listStatus/fs.delete throws IOException — e.g. NameNode unreachable, HDFS permission denied, snapshot on the path, or a filesystem RPC failure during cleanup of empty parent dirs.

Common situations: Coordinator issuing kill tasks while the HDFS cluster is degraded or in safe mode; HDFS permissions changed after segments were pushed; the segment directory is inside an HDFS snapshot (deletion forbidden); transient network partitions between Druid historical/coordinator nodes and the NameNode.

Related errors

• Couldn't delete segments from Azure. See the task logs for…
• Don't know how to load SCHEME
• Error loading [ ]
• No files found at [ ]
• [ ] is a child directory, skipping

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/c7099304cba3040c. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java:125

                  segmentParts[0],
                  segmentParts.length == 2 ? "" : segmentParts[1] + "_"
              )
          );
        }

        if (!fs.delete(segmentPath, false)) {
          throw new SegmentLoadingException("Unable to kill segment, failed to delete [%s]", segmentPath.toString());
        }

        // descriptor.json is a file to store segment metadata in deep storage. This file is deprecated and not stored
        // anymore, but we still delete them if exists.
        fs.delete(descriptorPath, false);

        removeEmptyParentDirectories(fs, segmentPath, segmentParts.length > 1 ? 2 : 3);
      }
    }
    catch (IOException e) {
      throw new SegmentLoadingException(e, "Unable to kill segment");
    }
  }

  @Override
  public void killAll() throws IOException
  {
    if (storageDirectory == null) {
      throw new ISE("Cannot delete all segment files since druid.storage.storageDirectory is not set.");
    }

    log.info("Deleting all segment files from hdfs dir [%s].", storageDirectory.toUri().toString());
    final FileSystem fs = storageDirectory.getFileSystem(config);
    fs.delete(storageDirectory, true);
  }

  private void removeEmptyParentDirectories(final FileSystem fs, final Path segmentPath, final int depth)
  {
    Path path = segmentPath;

View on GitHub (pinned to 9b90983fd2)