What causes "Failed to delete deep storage directory[ ]." in apache/druid?

Run `hdfs dfs -rm -r / ` manually as the Druid user to see HDFS's actual PermissionDenied/snapshot error.

apache/druid · error · IOException

Failed to delete deep storage directory[<dirToDelete>].

Error message

Failed to delete deep storage directory[<dirToDelete>].

What it means

HdfsDataSegmentKiller.killRecursively() throws a plain IOException when FileSystem.delete(dirToDelete, true) returns false, i.e. HDFS itself reported the recursive delete did not succeed even though the directory exists. The path being deleted is storageDirectory + relativePath (with ':' replaced by '_'); the delete returning false typically means permission denial or an HDFS-side constraint rather than a missing path (a missing path returns early).

Source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java:171

      log.makeAlert(e, "uncaught exception during segment killer").emit();
    }
  }

  @Override
  public void killRecursively(String relativePath) throws IOException
  {
    final Path dirToDelete = constructHdfsDeletePath(relativePath);
    if (dirToDelete == null) {
      return;
    }

    final FileSystem fs = dirToDelete.getFileSystem(config);
    if (!fs.exists(dirToDelete)) {
      return;
    }
    log.info("Deleting deep storage directory[%s]", dirToDelete);
    if (!fs.delete(dirToDelete, true)) {
      throw new IOException("Failed to delete deep storage directory[" + dirToDelete + "].");
    }
  }

  /**
   * Construct a path to delete from HDFS. Returns null if the path is invalid.
   * Replicates how {@link HdfsDataSegmentPusher#pushToPath} handles ':', by replacing that with '_'.
   */
  @Nullable
  private Path constructHdfsDeletePath(String relativePath)
  {
    if (Strings.isNullOrEmpty(relativePath)) {
      log.warn("Skipping deep storage directory kill: relative path is empty");
      return null;
    }
    if (relativePath.charAt(0) == '/') {
      log.warn("Skipping deep storage directory kill: relative path must not be absolute, got [%s]", relativePath);
      return null;
    }

View on GitHub (pinned to 9b90983fd2)

Solutions

Run `hdfs dfs -rm -r <storageDirectory>/<relativePath>` manually as the Druid user to see HDFS's actual PermissionDenied/snapshot error.
Grant the Druid service user write/delete permission (chown/chmod or dfs.namenode.acls) on the segment parent directory.
Check whether the directory is inside an HDFS snapshot and remove the snapshot entry if deletion is intended.
Ensure the NameNode is out of safe mode and retry the operation.
Verify the resolved path (storageDirectory + relativePath, ':' -> '_') is the one you intend to delete before adjusting permissions.

Example fix

// before: delete silently returns false due to ownership
// $ sudo -u druid hdfs dfs -rm -r /druid/segments/wikipedia
//   rm: Permission denied: user=druid, access=ALL, inode="/druid/segments/wikipedia":hdfs:supergroup:drwxr-xr-x
// after
// $ sudo -u hdfs hdfs dfs -chown -R druid /druid/segments/wikipedia
// $ sudo -u druid hdfs dfs -rm -r /druid/segments/wikipedia

Defensive patterns

Strategy: try-catch

Validate before calling

Path target = new Path(storageDir + "/" + relativePath.replace(':', '_'));
FileSystem fs = target.getFileSystem(config);
if (!fs.exists(target)) { return; } // nothing to delete
// optionally: fs.access(target, FsAction.WRITE) where available

Type guard

if (relativePath == null || relativePath.startsWith("/") || relativePath.contains("..")) { skip; }

Try / catch

try { killer.killRecursively(relPath); }
catch (IOException e) { log.error(e, "Recursive delete returned false for %s — check permissions/snapshots", relPath); }

Prevention

Ensure the Druid service user owns (or can modify) the entire segment tree it creates.
Do not snapshot deep-storage directories that must be deletable.
Confirm the computed path (with ':' -> '_') matches the pushed layout before deleting.

When it happens

Trigger: Calling killRecursively(relativePath) when fs.delete(..., true) returns false — commonly because the Druid user lacks delete permission on the directory or its parent, the path is inside an HDFS snapshot, NameNode is in safe mode, or an HA/retry layer swallowed the actual error and returned false.

Common situations: Dropping a datasource where the HDFS segment directory is owned by another user (e.g. created by a different Kerberos principal); segment directories under a snapshot path; quota or encryption-zone policies blocking deletion; delete racing with concurrent segment writes.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/2e1d59f42c037999. Report an issue: GitHub.