apache/druid · error · IOException

Unexpected subdirectory [%s]

Error message

Unexpected subdirectory [%s]

What it means

LocalDataSegmentPusher.pushNoZip throws IOException when it encounters a subdirectory inside the segment input directory. Segment directories are expected to be flat (index.drd, data files at top level); a nested directory means the segment layout is not what the no-zip pusher supports, so it refuses to push rather than producing a broken segment.

Source

Thrown at server/src/main/java/org/apache/druid/segment/loading/LocalDataSegmentPusher.java:146

  private DataSegment pushNoZip(final File inDir, final File outDir, final DataSegment baseSegment) throws IOException
  {
    final File tmpSegmentDir = new File(config.getStorageDirectory(), makeIntermediateDir());
    FileUtils.mkdirp(tmpSegmentDir);

    try {
      final File[] files = inDir.listFiles();
      if (files == null) {
        throw new IOE("Cannot list directory [%s]", inDir);
      }

      long size = 0;
      for (final File file : files) {
        if (file.isFile()) {
          size += file.length();
          FileUtils.linkOrCopy(file, new File(tmpSegmentDir, file.getName()));
        } else {
          // Segment directories are expected to be flat.
          throw new IOE("Unexpected subdirectory [%s]", file.getName());
        }
      }

      final File segmentDir = new File(outDir, INDEX_DIR);
      FileUtils.mkdirp(outDir);

      try {
        Files.move(tmpSegmentDir.toPath(), segmentDir.toPath(), StandardCopyOption.ATOMIC_MOVE);
      }
      catch (IOException e) {
        if (segmentDir.exists()) {
          // Move old directory out of the way, then try again. This makes the latest push win when we push to the
          // same directory twice, so behavior is compatible with the zip style of pushing.
          Files.move(
              segmentDir.toPath(),
              new File(outDir, StringUtils.format("%s_old_%s", INDEX_DIR, UUID.randomUUID())).toPath(),
              StandardCopyOption.ATOMIC_MOVE
          );

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Remove unexpected subdirectories from the segment output directory and re-push.
2. Ensure the ingestion task produces a flat segment directory (only files at top level).
3. Use zip push mode (druid.storage.type / pusher default) if nested layouts are unavoidable.
4. Check custom plugins/external scripts that write into the task work directory.

Example fix

// before: extra subdir left in segment dir
/segment-dir/index.drd
/segment-dir/data/000.smoosh  <-- 'data' is a subdirectory
// after: flatten
/segment-dir/index.drd
/segment-dir/000.smoosh

File[] files = inDir.listFiles();
if (files != null) {
  for (File f : files) {
    if (f.isDirectory()) throw new IllegalStateException("nested dir in segment output, flatten first: " + f);
  }
}

boolean isFlatSegmentDir(File dir) {
  File[] files = dir.listFiles();
  if (files == null) return false;
  return java.util.Arrays.stream(files).allMatch(File::isFile);
}

try {
  pusher.push(segment, outDir, false);
} catch (IOException e) {
  log.error(e, "segment dir layout invalid; flatten and retry");
  throw e;
}

When it happens

Trigger: pushToPath -> pushNoZip iterates inDir.listFiles() and finds file.isDirectory() true — a nested folder in the segment output.

Common situations: Custom extension or pre-processing step writing extra subdirectories (logs, smolder temp dirs) into the segment dir; users manually placing files in the task output; mixing uncompressed push with layouts produced by other tools.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Failed to rename [%s] to [%s]
• Cannot list directory [%s]
• RuntimeException wrapping underlying cause
• Failed to upload [%s] to [%s]
• Unable to load from local directory [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8e1b8fe415aacc39. Report an issue: GitHub.