apache/druid · warning · IOException

Thread interrupted. Couldn't delete all tasklogs.

Error message

Thread interrupted. Couldn't delete all tasklogs.

What it means

While deleting old task logs, killOlderThan() checks the thread's interrupt status after each deletion; if interrupted, it throws an IOException wrapping an InterruptedException saying not all task logs could be deleted. The loop keeps deleting until it notices the interrupt, then aborts so shutdown can proceed promptly.

Source

Thrown at extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/tasklog/HdfsTaskLogs.java:204

    FileSystem fs = taskLogDir.getFileSystem(hadoopConfig);
    if (fs.exists(taskLogDir)) {
      FileStatus taskLogFileStatus = fs.getFileStatus(taskLogDir);

      if (!taskLogFileStatus.isDirectory()) {
        throw new IOE("taskLogDir [%s] must be a directory.", taskLogDir);
      }

      RemoteIterator<LocatedFileStatus> iter = fs.listLocatedStatus(taskLogDir);
      while (iter.hasNext()) {
        LocatedFileStatus file = iter.next();
        if (file.getModificationTime() < timestamp) {
          Path p = file.getPath();
          log.info("Deleting hdfs task log [%s].", p.toUri().toString());
          fs.delete(p, true);
        }

        if (Thread.currentThread().isInterrupted()) {
          throw new IOException(
              new InterruptedException("Thread interrupted. Couldn't delete all tasklogs.")
          );
        }
      }
    }
  }

  @Override
  public void pushTaskPayload(String taskId, File taskPayloadFile) throws IOException
  {
    final Path path = getTaskPayloadFileFromId(taskId);
    log.info("Pushing payload for task[%s] to location[%s]", taskId, path);
    pushTaskFile(path, taskPayloadFile);
  }

  @Override
  public Optional<InputStream> streamTaskPayload(String taskId) throws IOException
  {

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Re-run the task-log cleanup after the interruption; deletion is idempotent
2. Avoid restarting/killing Druid services while cleanup is running
3. If cleanup repeatedly fails, batch smaller kills or run cleanup during low-load windows
4. Increase killTaskTimeout / scheduling window so large cleanups complete before interruption

Example fix

null

null

try {
  taskLogs.killOlderThan(cutoffMillis);
} catch (IOException e) {
  if (e.getCause() instanceof InterruptedException) {
    // interrupted mid-cleanup; safe to re-run, deletion is idempotent
  }
}

When it happens

Trigger: The thread running killOlderThan() is interrupted (task cancelled, service shutdown, killOldTaskLogs timed out) while HDFS delete operations are still pending for remaining log files.

Common situations: Overlord shutting down or a kill task timing out while a large backlog of old HDFS task logs is being purged; operator restarting services during a cleanup run.

Related errors

• taskLogDir [%s] must be a directory.
• Failed to upload [%s] to [%s]
• Failed to stream logs from: %s
• '%s' must be a string or an array of strings
• Only %s protocols are allowed

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/cf3f11e051231d5c. Report an issue: GitHub.