apache/druid · error · IllegalStateException

outDir[%s] must exist and be a directory

Error message

outDir[%s] must exist and be a directory

What it means

Precondition guard in the LZ4 directory decompression path: lz4DecompressDirectory streams an LZ4-compressed directory archive into an output directory that must already exist on disk, because the routine writes entries relative to it and never creates the root itself. It fires when callers (decompressDirectory) pass an outDir that does not exist or is a regular file rather than a directory; create the target directory before decompressing.

Source

Thrown at processing/src/main/java/org/apache/druid/utils/CompressionUtils.java:362

      // Copy file content to dataOut
      try (FileInputStream fileIn = new FileInputStream(file)) {
        ByteStreams.copy(fileIn, dataOut);
      }
    }

    dataOut.flush();
    lz4Out.finish();
    return totalSize;
  }

  /**
   * Decompresses LZ4-compressed directory archive
   */
  public static FileUtils.FileCopyResult lz4DecompressDirectory(InputStream in, File outDir) throws IOException
  {
    if (!(outDir.exists() && outDir.isDirectory())) {
      throw new ISE("outDir[%s] must exist and be a directory", outDir);
    }

    final LZ4BlockInputStream lz4In = new LZ4BlockInputStream(in);
    final DataInputStream dataIn = new DataInputStream(lz4In);

    final int fileCount = dataIn.readInt();
    final FileUtils.FileCopyResult result = new FileUtils.FileCopyResult();

    for (int i = 0; i < fileCount; i++) {
      final int fileNameLength = dataIn.readInt();
      final byte[] fileNameBytes = new byte[fileNameLength];
      dataIn.readFully(fileNameBytes);
      final String fileName = new String(fileNameBytes, StandardCharsets.UTF_8);

      final long fileSize = dataIn.readLong();

      // Write to file
      final File outFile = new File(outDir, fileName);

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Create the directory before calling: `Files.mkdirs(outDir)` or `outDir.mkdirs()` and verify `outDir.isDirectory()`.
2. Check the configured path for typos and ensure the Druid process user can create/write there.
3. If the directory was deleted by cleanup, recreate it before decompression.

Example fix

// before
CompressionUtils.lz4DecompressDirectory(in, new File("/tmp/out"));
// after
final File outDir = new File("/tmp/out");
if (!outDir.exists() && !outDir.mkdirs()) {
  throw new IOException("Failed to create outDir: " + outDir);
}
CompressionUtils.lz4DecompressDirectory(in, outDir);

final File outDir = new File(dest);
if (!outDir.isDirectory() && !outDir.mkdirs()) {
  throw new IOException("Cannot create output directory: " + outDir);
}

static boolean isValidOutDir(File f) {
  return f != null && f.exists() && f.isDirectory();
}

try {
  CompressionUtils.lz4DecompressDirectory(in, outDir);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("must exist and be a directory")) {
    outDir.mkdirs();
    // retry once
  }
}

When it happens

Trigger: Calling decompressDirectory (LZ4 mode) with an outDir File that was never mkdir'd, was deleted earlier in the flow, or is a plain file.

Common situations: Fresh worker nodes where the local segment cache dir hasn't been created yet, typo'd outdir path in task config, or code that assumes decompress creates the directory.

Related errors

• Cannot list files in directory[%s]
• Cannot list directory [%s]
• Could not close channel for level [%d] and rank [%d]
• Unable to close channel for name :
• Path [%s] is not writable, check permissions

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/bfa3332a91643a55. Report an issue: GitHub.