apache/druid · error · IllegalStateException

Task [ ] has been shut down

Error message

Task [%s] has been shut down

What it means

KubernetesTaskRunner.doTask() throws this IllegalStateException when the task's work item is absent from the runner's in-memory tasks map - the task was already completed/cancelled and removed (shut down), so its KubernetesPeonLifecycle no longer exists. There is no peon process left to run or join.

Solutions

1. Check the task is still registered (not cancelled/stopped) before calling runTask/joinTask
2. Treat the ISE as 'task no longer tracked' and re-query task status from the Overlord/task storage rather than retrying
3. Ensure stop() is not called while tasks are expected to be runnable
4. On runner restart, resubmit or re-track tasks before invoking doTask

Example fix

// before
TaskStatus status = taskRunner.runTask(task, location); // ISE if removed
// after
if (taskRunner.getKnownTasks().contains(task.getId())) {
  TaskStatus status = taskRunner.runTask(task, location);
} else {
  // task was shut down; fetch status from task storage
}

if (!taskRunner.getKnownTasks().contains(task.getId())) { /* task already shut down; re-query status */ }

try {
  return taskRunner.runTask(task, location);
} catch (IllegalStateException e) {
  // work item removed: task completed/cancelled; fall back to task storage
  return taskStorage.getStatus(task.getId());
}

When it happens

Trigger: runTask()/joinTask() invoked for a task whose work item was removed by stop(), task cancellation, or a previous doTask completion; a race where the peon finishes and cleanup removes the work item while the caller still tries to run/join it.

Common situations: Overlord querying status of a task just as the runner shuts down; duplicate task submission where one path already consumed and removed the item; runner restart clearing the tasks map while the Overlord still references task IDs.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Listener [ ] already registered
• can't start.
• can't start.
• can't stop.
• can't stop.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/6116e970fb660409. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java:269

    return doTask(task, true);
  }

  private TaskStatus joinTask(Task task)
  {
    return doTask(task, false);
  }

  @VisibleForTesting
  protected TaskStatus doTask(Task task, boolean run)
  {
    try {
      KubernetesPeonLifecycle peonLifecycle;

      synchronized (tasks) {
        KubernetesWorkItem workItem = tasks.get(task.getId());

        if (workItem == null) {
          throw new ISE("Task [%s] has been shut down", task.getId());
        }

        peonLifecycle = workItem.getPeonLifeycle();
      }

      TaskStatus taskStatus;
      if (run) {
        taskStatus = peonLifecycle.run(
            adapter.fromTask(task),
            config.getTaskLaunchTimeout().toStandardDuration().getMillis(),
            config.getTaskTimeout().toStandardDuration().getMillis(),
            adapter.shouldUseDeepStorageForTaskPayload(task)
        );
      } else {
        taskStatus = peonLifecycle.join(
            config.getTaskTimeout().toStandardDuration().getMillis()
        );
      }

View on GitHub (pinned to 9b90983fd2)