{"record":{"id":"df0be267653839c9","repo":"apache/druid","slug":"error-while-cleaning-up-durable-storage-path-s","errorCode":null,"errorMessage":"Error while cleaning up durable storage path[%s].","messagePattern":"Error while cleaning up durable storage path\\[(.+?)\\]\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java","lineNumber":989,"sourceCode":"   */\n  private void removeStageDurableStorageOutput(final StageId stageId)\n  {\n    // One caveat with this approach is that in case of a worker crash, while the MM/Indexer systems will delete their\n    // temp directories where intermediate results were stored, it won't be the case for the external storage.\n    // Therefore, the logic for cleaning the stage output in case of a worker/machine crash has to be external.\n    // We currently take care of this in the controller.\n    final String folderName = DurableStorageUtils.getTaskIdOutputsFolderName(\n        task.getControllerTaskId(),\n        stageId.getStageNumber(),\n        task.getWorkerNumber(),\n        context.workerId()\n    );\n    try {\n      MSQTasks.makeStorageConnector(context.injector()).deleteRecursively(folderName);\n    }\n    catch (Exception e) {\n      // If an error is thrown while cleaning up a file, log it and try to continue with the cleanup\n      log.warn(e, \"Error while cleaning up durable storage path[%s].\", folderName);\n    }\n  }\n\n  private StageOutputHolder getOrCreateStageOutputHolder(final StageId stageId, final int partitionNumber)\n  {\n    return stageOutputs\n        .computeIfAbsent(stageId, ignored1 -> new ConcurrentHashMap<>())\n        .computeIfAbsent(partitionNumber, ignored -> new StageOutputHolder(getWireTransferableContext()));\n  }\n\n  /**\n   * Retrieve {@link WireTransferableContext} from our injector.\n   */\n  private WireTransferableContext getWireTransferableContext()\n  {\n    return context.injector().getInstance(WireTransferableContext.class);\n  }\n","sourceCodeStart":971,"sourceCodeEnd":1007,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java#L971-L1007","documentation":"WorkerImpl cleans up the task's durable storage directory when a query finishes. If MSQTasks.makeStorageConnector(context.injector()).deleteRecursively(folderName) throws, the code logs a warning and continues cleanup rather than failing the task. This is intentionally non-fatal: leftover durable storage files do not affect query results, only disk usage.","triggerScenarios":"The durable storage connector (local file system or cloud) fails to delete the task's folderName directory recursively, e.g. permissions issues, files locked by concurrent tasks, cloud-provider transient errors, or the directory already removed by another process.","commonSituations":"Running multiple MSQ tasks sharing the same durable storage location with insufficient permissions; S3/GCS eventual-consistency or throttling errors during deleteRecursively; task crash leaving partially-deleted directories that later cleanup cannot fully remove; disk state changed externally between query finish and cleanup.","solutions":["Check permissions on the durable storage directory (druid.indexer.task.durableStorageDirectory / local path or cloud bucket) and grant the task user delete access","Verify the durable storage connector configuration (type, bucket/prefix) points at a location owned by this cluster only, not shared with other tasks","Look at the nested exception `e` in the log line for the root cause (e.g. 403, NoSuchKey, IOException) and fix that specific issue","Manually delete orphaned durable storage directories; they do not affect correctness, only disk usage","Retry the workload; this is often a transient cloud-storage error and cleanup is best-effort"],"exampleFix":"// before: shared durable storage with conflicting permissions\n// druid.storage.type=s3, bucket shared with other teams, IAM lacks s3:DeleteObject\n// after: dedicated bucket/prefix with task-role granted delete\n// durableStorage:\n//   type: local\n//   storageDirectory: /var/druid/durable-storage  # owned by the druid user","handlingStrategy":"validation","validationCode":"// Verify durable storage path is writable before submitting task\njava.nio.file.Path dir = java.nio.file.Paths.get(durableStorageDir);\nif (!java.nio.file.Files.isDirectory(dir) || !java.nio.file.Files.isWritable(dir)) {\n  throw new IllegalStateException(\"Durable storage dir not writable: \" + dir);\n}","typeGuard":null,"tryCatchPattern":"// Non-fatal by design; monitor logs\ntry {\n  cleanupDurableStorage(folderName);\n} catch (Exception e) {\n  log.warn(e, \"Cleanup of durable storage [%s] failed; continuing\", folderName);\n}","preventionTips":["Give the task user exclusive write/delete access to the durable storage directory","Do not share a durable storage prefix across clusters or concurrent task types","Schedule periodic orphan-directory sweeps since cleanup is best-effort","Check cloud storage IAM policies include delete permissions on the prefix"],"tags":["cleanup","storage","msq","best-effort"],"backgroundTag":"file-write-permission-denied","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}