apache/druid · error · IOE

taskLogDir [%s] must be a directory.

Error message

taskLogDir [%s] must be a directory.

What it means

HdfsTaskLogs.killOlderThan() validates that the configured task-log directory is a directory before iterating over old task logs. If the path exists in HDFS but is a regular file, an IOE is thrown. This prevents accidental deletion attempts against a wrongly configured path.

Source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/tasklog/HdfsTaskLogs.java:191

  @Override
  public void killAll() throws IOException
  {
    log.info("Deleting all task logs from hdfs dir [%s].", config.getDirectory());
    Path taskLogDir = new Path(config.getDirectory());
    FileSystem fs = taskLogDir.getFileSystem(hadoopConfig);
    fs.delete(taskLogDir, true);
  }

  @Override
  public void killOlderThan(long timestamp) throws IOException
  {
    Path taskLogDir = new Path(config.getDirectory());
    FileSystem fs = taskLogDir.getFileSystem(hadoopConfig);
    if (fs.exists(taskLogDir)) {
      FileStatus taskLogFileStatus = fs.getFileStatus(taskLogDir);

      if (!taskLogFileStatus.isDirectory()) {
        throw new IOE("taskLogDir [%s] must be a directory.", taskLogDir);
      }

      RemoteIterator<LocatedFileStatus> iter = fs.listLocatedStatus(taskLogDir);
      while (iter.hasNext()) {
        LocatedFileStatus file = iter.next();
        if (file.getModificationTime() < timestamp) {
          Path p = file.getPath();
          log.info("Deleting hdfs task log [%s].", p.toUri().toString());
          fs.delete(p, true);
        }

        if (Thread.currentThread().isInterrupted()) {
          throw new IOException(
              new InterruptedException("Thread interrupted. Couldn't delete all tasklogs.")
          );
        }
      }
    }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Fix druid.indexer.logs.directory in the middleManager/overlord config to point at a directory, e.g. hdfs://namenode/druid/tasklogs
2. Check what the path contains: hdfs dfs -ls <dir>; remove or rename the file if it was created by mistake
3. Restart the affected Druid services after correcting the config

Example fix

// before
druid.indexer.logs.directory=hdfs://nn/druid/tasklogs/current.log
// after
druid.indexer.logs.directory=hdfs://nn/druid/tasklogs/

Path dir = new Path(config.getDirectory());
FileSystem fs = dir.getFileSystem(hadoopConfig);
if (!fs.exists(dir) || !fs.getFileStatus(dir).isDirectory()) {
  throw new IllegalArgumentException("taskLogDir must be a directory: " + dir);
}

null

try {
  taskLogs.killOlderThan(cutoffMillis);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("taskLogDir")) {
    // fix druid.indexer.logs.directory config
  }
  throw e;
}

When it happens

Trigger: killOlderThan(timestamp) finds fs.exists(taskLogDir) true but fs.getFileStatus(taskLogDir).isDirectory() false — i.e. config.getDirectory() points at a file.

Common situations: druid.indexer.logs.directory mistakenly set to a file path; someone overwrote/created a file at the directory path in HDFS; copy-paste typo in the task-log config.

Related errors

• Cannot delete all segment files since druid.storage.storageD
• Thread interrupted. Couldn't delete all tasklogs.
• taskLogDir [%s] must be a directory.
• Skipping deep storage directory kill: storage directory not
• The gRPC query server requires either a Basic or Anonymous a

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/723b1ce8e17764fe. Report an issue: GitHub.