apache/druid · error · IllegalStateException

K8s pod for the task[%s] appeared and disappeared. It can ha

Error message

K8s pod for the task[%s] appeared and disappeared. It can happen if the task was canceled

What it means

Thrown by KubernetesPeonClient.launchPeonJobAndWaitForStart when the Kubernetes Job for a task was submitted successfully and waitUntilPeonPodCreatedAndReady returned null, meaning the pod was expected to exist but could no longer be found after the wait window. Druid treats this as an internal state error because a freshly created job's pod should be present; disappearance usually means the job/pod was deleted, most commonly by task cancellation.

Source

Thrown at extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java:96

    this.debugJobs = debugJobs;
    this.emitter = emitter;
  }

  public Pod launchPeonJobAndWaitForStart(Job job, Task task, long howLong, TimeUnit timeUnit) throws IllegalStateException
  {
    long start = System.currentTimeMillis();
    // launch job
    return clientApi.executeRequest(client -> {
      String jobName = job.getMetadata().getName();

      log.info("Submitting job[%s] for task[%s].", jobName, task.getId());
      createK8sJobWithRetries(job);
      log.info("Submitted job[%s] for task[%s]. Waiting for POD to launch.", jobName, task.getId());

      Pod result = waitUntilPeonPodCreatedAndReady(jobName, howLong, timeUnit);

      if (result == null) {
        throw new ISE("K8s pod for the task[%s] appeared and disappeared. It can happen if the task was canceled", task.getId());
      }
      log.info("Pod for job[%s] is in state[%s] for task[%s].", jobName, result.getStatus().getPhase(), task.getId());
      long duration = System.currentTimeMillis() - start;
      emitK8sPodMetrics(task, "k8s/peon/startup/time", duration);
      return result;
    });
  }

  /**
   * Waits until a pod for the given job is created and ready to be monitored.
   * <p>
   * A pod can appear and dissapear in some cases, such as the task being canceled. In this case, null is returned and
   * the caller should handle accordingly.
   * </p>
   *
   * @param jobName  the name of the job whose pod we're waiting for
   * @param howLong  the maximum time to wait
   * @param timeUnit the time unit for the timeout

View on GitHub (pinned to 9b90983fd2)

Solutions

Check whether the task was canceled (Overlord logs / task API); a cancellation racing startup is expected behavior
Increase the peon pod start timeout (e.g. druid.indexer.runner.k8s.peonTimeout / task start wait settings) if pods are slow to schedule
Investigate what removed the pod: job events via kubectl describe job, eviction notices, autoscaler or cleanup jobs
Re-run/resubmit the task; the condition is usually transient

Example fix

// no code fix; operational — check cancellation and pod events
kubectl describe job <peon-job-name> -n <namespace>
kubectl get events -n <namespace> | grep <pod-name>

Defensive patterns

Strategy: retry

Validate before calling

// Before submission, ensure the task is not in a cancelled state
TaskInfo info = overlordClient.getTaskInfo(taskId);
if (info != null && info.getStatusCode() == TaskState.CANCELLED) {
    return; // do not wait for a pod that will be deleted
}

Try / catch

try {
    Pod pod = kubernetesPeonClient.launchPeonJobAndWaitForStart(job, task, timeout, unit);
} catch (IllegalStateException e) {
    if (e.getMessage().contains("appeared and disappeared")) {
        // check if task was canceled; if so, treat as expected shutdown, else retry once
        if (!overlordClient.getTaskInfo(task.getId()).isCancelled()) {
            // resubmit or alert
        }
    } else {
        throw e;
    }
}

Prevention

Increase peon pod startup timeout in slow-scheduling clusters
Avoid aggressive external cleanup jobs that delete young pods or jobs
Watch for task cancellations racing pod startup; they are the main benign cause

When it happens

Trigger: createK8sJobWithRetries succeeds, then waitUntilPeonPodCreatedAndReady times out or observes the pod vanish (returns null) within the configured wait duration; the task's cancellation path or an external controller deleted the job/pod in the meantime.

Common situations: User or Overlord cancels a task while the peon pod is still starting; external cleanup scripts or namespace-wide eviction deleting young pods; very short peon start timeouts combined with slow pod scheduling; cluster autoscaler scaling away nodes before pods bind.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a0bf24edab36cf2b. Report an issue: GitHub.