{"record":{"id":"cf3f11e051231d5c","repo":"apache/druid","slug":"thread-interrupted-couldn-t-delete-all-tasklogs","errorCode":null,"errorMessage":"Thread interrupted. Couldn't delete all tasklogs.","messagePattern":"Thread interrupted\\. Couldn't delete all tasklogs\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/tasklog/HdfsTaskLogs.java","lineNumber":204,"sourceCode":"    FileSystem fs = taskLogDir.getFileSystem(hadoopConfig);\n    if (fs.exists(taskLogDir)) {\n      FileStatus taskLogFileStatus = fs.getFileStatus(taskLogDir);\n\n      if (!taskLogFileStatus.isDirectory()) {\n        throw new IOE(\"taskLogDir [%s] must be a directory.\", taskLogDir);\n      }\n\n      RemoteIterator<LocatedFileStatus> iter = fs.listLocatedStatus(taskLogDir);\n      while (iter.hasNext()) {\n        LocatedFileStatus file = iter.next();\n        if (file.getModificationTime() < timestamp) {\n          Path p = file.getPath();\n          log.info(\"Deleting hdfs task log [%s].\", p.toUri().toString());\n          fs.delete(p, true);\n        }\n\n        if (Thread.currentThread().isInterrupted()) {\n          throw new IOException(\n              new InterruptedException(\"Thread interrupted. Couldn't delete all tasklogs.\")\n          );\n        }\n      }\n    }\n  }\n\n  @Override\n  public void pushTaskPayload(String taskId, File taskPayloadFile) throws IOException\n  {\n    final Path path = getTaskPayloadFileFromId(taskId);\n    log.info(\"Pushing payload for task[%s] to location[%s]\", taskId, path);\n    pushTaskFile(path, taskPayloadFile);\n  }\n\n  @Override\n  public Optional<InputStream> streamTaskPayload(String taskId) throws IOException\n  {","sourceCodeStart":186,"sourceCodeEnd":222,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/tasklog/HdfsTaskLogs.java#L186-L222","documentation":"While deleting old task logs, killOlderThan() checks the thread's interrupt status after each deletion; if interrupted, it throws an IOException wrapping an InterruptedException saying not all task logs could be deleted. The loop keeps deleting until it notices the interrupt, then aborts so shutdown can proceed promptly.","triggerScenarios":"The thread running killOlderThan() is interrupted (task cancelled, service shutdown, killOldTaskLogs timed out) while HDFS delete operations are still pending for remaining log files.","commonSituations":"Overlord shutting down or a kill task timing out while a large backlog of old HDFS task logs is being purged; operator restarting services during a cleanup run.","solutions":["Re-run the task-log cleanup after the interruption; deletion is idempotent","Avoid restarting/killing Druid services while cleanup is running","If cleanup repeatedly fails, batch smaller kills or run cleanup during low-load windows","Increase killTaskTimeout / scheduling window so large cleanups complete before interruption"],"exampleFix":"null","handlingStrategy":"retry","validationCode":"null","typeGuard":null,"tryCatchPattern":"try {\n  taskLogs.killOlderThan(cutoffMillis);\n} catch (IOException e) {\n  if (e.getCause() instanceof InterruptedException) {\n    // interrupted mid-cleanup; safe to re-run, deletion is idempotent\n  }\n}","preventionTips":["Re-run cleanup after interruption; HDFS deletes are idempotent","Schedule log-kill tasks during low-traffic windows","Avoid service restarts while cleanup is in progress","Size cleanup runs so they finish well before timeouts"],"tags":["hdfs","task-logs","interruption","cleanup"],"backgroundTag":"operation-interrupted","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}