apache/druid · warning

Error while cleaning up durable storage path[%s].

Error message

Error while cleaning up durable storage path[%s].

What it means

WorkerImpl cleans up the task's durable storage directory when a query finishes. If MSQTasks.makeStorageConnector(context.injector()).deleteRecursively(folderName) throws, the code logs a warning and continues cleanup rather than failing the task. This is intentionally non-fatal: leftover durable storage files do not affect query results, only disk usage.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java:989

   */
  private void removeStageDurableStorageOutput(final StageId stageId)
  {
    // One caveat with this approach is that in case of a worker crash, while the MM/Indexer systems will delete their
    // temp directories where intermediate results were stored, it won't be the case for the external storage.
    // Therefore, the logic for cleaning the stage output in case of a worker/machine crash has to be external.
    // We currently take care of this in the controller.
    final String folderName = DurableStorageUtils.getTaskIdOutputsFolderName(
        task.getControllerTaskId(),
        stageId.getStageNumber(),
        task.getWorkerNumber(),
        context.workerId()
    );
    try {
      MSQTasks.makeStorageConnector(context.injector()).deleteRecursively(folderName);
    }
    catch (Exception e) {
      // If an error is thrown while cleaning up a file, log it and try to continue with the cleanup
      log.warn(e, "Error while cleaning up durable storage path[%s].", folderName);
    }
  }

  private StageOutputHolder getOrCreateStageOutputHolder(final StageId stageId, final int partitionNumber)
  {
    return stageOutputs
        .computeIfAbsent(stageId, ignored1 -> new ConcurrentHashMap<>())
        .computeIfAbsent(partitionNumber, ignored -> new StageOutputHolder(getWireTransferableContext()));
  }

  /**
   * Retrieve {@link WireTransferableContext} from our injector.
   */
  private WireTransferableContext getWireTransferableContext()
  {
    return context.injector().getInstance(WireTransferableContext.class);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Check permissions on the durable storage directory (druid.indexer.task.durableStorageDirectory / local path or cloud bucket) and grant the task user delete access
2. Verify the durable storage connector configuration (type, bucket/prefix) points at a location owned by this cluster only, not shared with other tasks
3. Look at the nested exception `e` in the log line for the root cause (e.g. 403, NoSuchKey, IOException) and fix that specific issue
4. Manually delete orphaned durable storage directories; they do not affect correctness, only disk usage
5. Retry the workload; this is often a transient cloud-storage error and cleanup is best-effort

Example fix

// before: shared durable storage with conflicting permissions
// druid.storage.type=s3, bucket shared with other teams, IAM lacks s3:DeleteObject
// after: dedicated bucket/prefix with task-role granted delete
// durableStorage:
//   type: local
//   storageDirectory: /var/druid/durable-storage  # owned by the druid user

// Verify durable storage path is writable before submitting task
java.nio.file.Path dir = java.nio.file.Paths.get(durableStorageDir);
if (!java.nio.file.Files.isDirectory(dir) || !java.nio.file.Files.isWritable(dir)) {
  throw new IllegalStateException("Durable storage dir not writable: " + dir);
}

// Non-fatal by design; monitor logs
try {
  cleanupDurableStorage(folderName);
} catch (Exception e) {
  log.warn(e, "Cleanup of durable storage [%s] failed; continuing", folderName);
}

When it happens

Trigger: The durable storage connector (local file system or cloud) fails to delete the task's folderName directory recursively, e.g. permissions issues, files locked by concurrent tasks, cloud-provider transient errors, or the directory already removed by another process.

Common situations: Running multiple MSQ tasks sharing the same durable storage location with insufficient permissions; S3/GCS eventual-consistency or throttling errors during deleteRecursively; task crash leaving partially-deleted directories that later cleanup cannot fully remove; disk state changed externally between query finish and cleanup.

Understand the failure class

Background: "Permission denied" / "Failed to write" file errors: why a library can't write its files to disk (EACCES, EPERM, ENOSPC) and how to fix them — this error's family across 43 libraries.

Related errors

• Error while cleaning up temporary files at path[%s]. Skippin
• NotEnoughTemporaryStorageFault
• Thread interrupted. Couldn't delete all tasklogs.
• Exception while closing watch.
• Failed to get object summaries from S3 bucket[%s], prefix[%s

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/df0be267653839c9. Report an issue: GitHub.