apache/druid · warning

Skipping deep storage directory kill: invalid path

Error message

Skipping deep storage directory kill: invalid path[%s]

What it means

constructHdfsDeletePath splits the relative path on '/' and rejects it if any component is empty (consecutive slashes, leading/trailing slash) or equals '..' (parent-directory traversal). This prevents accidentally deleting directories outside the intended segment location. It logs this warning and returns null so the kill is skipped.

Solutions

1. Normalize the segment path in the metadata store (collapse duplicate slashes, remove '..')
2. Re-push the affected segments to get clean canonical paths
3. If the segment truly can't be fixed, delete the directory manually with hdfs dfs -rm -r after verifying the target

Example fix

// before
"path": "datasource//2019-01-01T00:00:00.000Z_../index.zip"
// after
"path": "datasource/2019-01-01T00:00:00.000Z_2019-01-02T00:00:00.000Z/2020-01-01T00:00:00.000Z/0/index.zip"

boolean valid = java.util.Arrays.stream(segmentPath.split("/", -1)).noneMatch(s -> s.isEmpty() || s.equals(".."));
if (!valid) { throw new IllegalArgumentException("invalid segment path: " + segmentPath); }

static boolean isSafeRelativePath(String p) {
  if (p == null) return false;
  for (String s : p.split("/", -1)) { if (s.isEmpty() || s.equals("..")) return false; }
  return true;
}

When it happens

Trigger: A DataSegment relative path like 'a//b', 'a/', or containing '..' segments is passed to dirToDelete during kill.

Common situations: Hand-edited segment loadSpecs, buggy custom deep-storage adapters producing double slashes, or malicious/corrupt metadata rows containing '..' traversal.

Understand the failure class

Background: Path traversal blocked: "path escapes the workspace" and "outside site root" errors when a path will not stay inside its allowed directory — this error's family across 26 libraries.

Related errors

• Skipping deep storage directory kill: backslash not allowed…
• Skipping deep storage directory kill: relative path is empty
• Skipping deep storage directory kill: relative path must…
• Segment path [ ] does not exist
• Skipping deep storage directory kill: storage directory not…

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a24ea3860362aadd. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/HdfsDataSegmentKiller.java:196

   */
  @Nullable
  private Path constructHdfsDeletePath(String relativePath)
  {
    if (Strings.isNullOrEmpty(relativePath)) {
      log.warn("Skipping deep storage directory kill: relative path is empty");
      return null;
    }
    if (relativePath.charAt(0) == '/') {
      log.warn("Skipping deep storage directory kill: relative path must not be absolute, got [%s]", relativePath);
      return null;
    }
    if (relativePath.indexOf('\\') >= 0) {
      log.warn("Skipping deep storage directory kill: backslash not allowed in path [%s]", relativePath);
      return null;
    }
    for (String segment : StringUtils.splitPreserveAllTokens(relativePath, '/')) {
      if (segment.isEmpty() || "..".equals(segment)) {
        log.warn("Skipping deep storage directory kill: invalid path[%s]", relativePath);
        return null;
      }
    }

    if (storageDirectory == null) {
      log.warn("Skipping deep storage directory kill: storage directory not configured");
      return null;
    }

    final String hdfsRelativePath = relativePath.replace(':', '_');
    final String storageDirectoryString = storageDirectory.toString();
    final String sep = storageDirectoryString.endsWith(Path.SEPARATOR) ? "" : Path.SEPARATOR;
    return new Path(storageDirectoryString + sep + hdfsRelativePath);
  }
}

View on GitHub (pinned to 9b90983fd2)