apache/druid · error · IOException

Cannot list files in directory[%s]

Error message

Cannot list files in directory[%s]

What it means

Thrown by CompressionUtils.lz4CompressDirectory when File.listFiles() returns null for the directory being LZ4-compressed. Java's listFiles() returns null when the File is not a directory or cannot be read (missing, permission denied, or an I/O error). Druid throws this instead of a cryptic NPE so the caller knows the directory could not be enumerated.

Source

Thrown at processing/src/main/java/org/apache/druid/utils/CompressionUtils.java:315

   */
  public static long lz4CompressDirectory(File directory, OutputStream out) throws IOException
  {
    if (!directory.isDirectory()) {
      throw new IOE("directory[%s] is not a directory", directory);
    }

    // Use fast compressor for better performance (lower CPU, faster compression)
    final LZ4BlockOutputStream lz4Out = new LZ4BlockOutputStream(
        out,
        64 * 1024, // Block size
        LZ4Factory.fastestInstance().fastCompressor()
    );
    // Use DataOutputStream for structured writing
    final DataOutputStream dataOut = new DataOutputStream(lz4Out);
    final File[] files = directory.listFiles();

    if (files == null) {
      throw new IOE("Cannot list files in directory[%s]", directory);
    }

    // Sort for consistency
    final File[] sortedFiles = Arrays.stream(files).sorted().toArray(File[]::new);

    dataOut.writeInt(sortedFiles.length);

    long totalSize = 0;

    for (File file : sortedFiles) {
      if (file.isDirectory()) {
        continue; // Skip subdirectories like ZIP does
      }

      log.debug("Compressing file[%s] with size[%,d]. Total size so far[%,d]", file, file.length(), totalSize);

      final String fileName = file.getName();
      final byte[] fileNameBytes = fileName.getBytes(StandardCharsets.UTF_8);

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify the directory exists and is readable by the Druid process user before calling compressDirectory (e.g. `File dir = new File(path); if (!dir.isDirectory()) throw ...;`).
2. Check filesystem permissions (ls -ld) and fix ownership/ACLs on the directory.
3. If the path should exist, check whether another process (task cleanup, tmpwatch) deleted it; re-create or restart the task.
4. Pass a File that actually points to a directory, not a file path.

Example fix

// before
CompressionUtils.lz4CompressDirectory(..., new File("/tmp/segments"), ...);
// after
final File dir = new File("/tmp/segments");
if (!dir.isDirectory()) {
  throw new IllegalStateException("Not a readable directory: " + dir);
}
CompressionUtils.lz4CompressDirectory(..., dir, ...);

final File dir = new File(path);
if (!dir.isDirectory()) {
  throw new IllegalArgumentException("Not a readable directory: " + dir);
}
if (!dir.canRead()) {
  throw new IOException("No read permission on: " + dir);
}

static boolean isListableDirectory(File f) {
  return f != null && f.isDirectory() && f.canRead();
}

try {
  CompressionUtils.compressDirectory(...);
} catch (IOException e) {
  if (e.getMessage().contains("Cannot list files in directory")) {
    // recreate/repair directory or fail task with clear message
  }
  throw e;
}

When it happens

Trigger: Calling compressDirectory (LZ4 mode) with a directory path that does not exist, points to a regular file instead of a directory, or the process lacks read permission on the directory.

Common situations: Task working directories deleted underneath a running ingestion task (container cleanup), wrong path in loadSpec/durable storage config, running Druid as a user without permissions on the segment cache dir, or a race where the directory is removed between existence check and listFiles().

Related errors

• outDir[%s] must exist and be a directory
• Cannot list directory [%s]
• Could not close channel for level [%d] and rank [%d]
• Unable to close channel for name :
• Path [%s] is not writable, check permissions

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/61968732f06c38a4. Report an issue: GitHub.