{"record":{"id":"6116e970fb660409","repo":"apache/druid","slug":"task-s-has-been-shut-down","errorCode":null,"errorMessage":"Task [%s] has been shut down","messagePattern":"Task \\[(.+?)\\] has been shut down","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java","lineNumber":269,"sourceCode":"    return doTask(task, true);\n  }\n\n  private TaskStatus joinTask(Task task)\n  {\n    return doTask(task, false);\n  }\n\n  @VisibleForTesting\n  protected TaskStatus doTask(Task task, boolean run)\n  {\n    try {\n      KubernetesPeonLifecycle peonLifecycle;\n\n      synchronized (tasks) {\n        KubernetesWorkItem workItem = tasks.get(task.getId());\n\n        if (workItem == null) {\n          throw new ISE(\"Task [%s] has been shut down\", task.getId());\n        }\n\n        peonLifecycle = workItem.getPeonLifeycle();\n      }\n\n      TaskStatus taskStatus;\n      if (run) {\n        taskStatus = peonLifecycle.run(\n            adapter.fromTask(task),\n            config.getTaskLaunchTimeout().toStandardDuration().getMillis(),\n            config.getTaskTimeout().toStandardDuration().getMillis(),\n            adapter.shouldUseDeepStorageForTaskPayload(task)\n        );\n      } else {\n        taskStatus = peonLifecycle.join(\n            config.getTaskTimeout().toStandardDuration().getMillis()\n        );\n      }","sourceCodeStart":251,"sourceCodeEnd":287,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java#L251-L287","documentation":"KubernetesTaskRunner.doTask() throws this IllegalStateException when the task's work item is absent from the runner's in-memory tasks map - the task was already completed/cancelled and removed (shut down), so its KubernetesPeonLifecycle no longer exists. There is no peon process left to run or join.","triggerScenarios":"runTask()/joinTask() invoked for a task whose work item was removed by stop(), task cancellation, or a previous doTask completion; a race where the peon finishes and cleanup removes the work item while the caller still tries to run/join it.","commonSituations":"Overlord querying status of a task just as the runner shuts down; duplicate task submission where one path already consumed and removed the item; runner restart clearing the tasks map while the Overlord still references task IDs.","solutions":["Check the task is still registered (not cancelled/stopped) before calling runTask/joinTask","Treat the ISE as 'task no longer tracked' and re-query task status from the Overlord/task storage rather than retrying","Ensure stop() is not called while tasks are expected to be runnable","On runner restart, resubmit or re-track tasks before invoking doTask"],"exampleFix":"// before\nTaskStatus status = taskRunner.runTask(task, location); // ISE if removed\n// after\nif (taskRunner.getKnownTasks().contains(task.getId())) {\n  TaskStatus status = taskRunner.runTask(task, location);\n} else {\n  // task was shut down; fetch status from task storage\n}","handlingStrategy":"try-catch","validationCode":"if (!taskRunner.getKnownTasks().contains(task.getId())) { /* task already shut down; re-query status */ }","typeGuard":null,"tryCatchPattern":"try {\n  return taskRunner.runTask(task, location);\n} catch (IllegalStateException e) {\n  // work item removed: task completed/cancelled; fall back to task storage\n  return taskStorage.getStatus(task.getId());\n}","preventionTips":["Check task liveness before run/join","Do not call stop() on the runner while tasks are in flight","Handle runner restarts by re-tracking tasks before use"],"tags":["kubernetes","task-runner","illegal-state","race-condition"],"backgroundTag":"resource-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}