{"record":{"id":"a0bf24edab36cf2b","repo":"apache/druid","slug":"k8s-pod-for-the-task-s-appeared-and-disappeared","errorCode":null,"errorMessage":"K8s pod for the task[%s] appeared and disappeared. It can happen if the task was canceled","messagePattern":"K8s pod for the task\\[(.+?)\\] appeared and disappeared\\. It can happen if the task was canceled","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java","lineNumber":96,"sourceCode":"    this.debugJobs = debugJobs;\n    this.emitter = emitter;\n  }\n\n  public Pod launchPeonJobAndWaitForStart(Job job, Task task, long howLong, TimeUnit timeUnit) throws IllegalStateException\n  {\n    long start = System.currentTimeMillis();\n    // launch job\n    return clientApi.executeRequest(client -> {\n      String jobName = job.getMetadata().getName();\n\n      log.info(\"Submitting job[%s] for task[%s].\", jobName, task.getId());\n      createK8sJobWithRetries(job);\n      log.info(\"Submitted job[%s] for task[%s]. Waiting for POD to launch.\", jobName, task.getId());\n\n      Pod result = waitUntilPeonPodCreatedAndReady(jobName, howLong, timeUnit);\n\n      if (result == null) {\n        throw new ISE(\"K8s pod for the task[%s] appeared and disappeared. It can happen if the task was canceled\", task.getId());\n      }\n      log.info(\"Pod for job[%s] is in state[%s] for task[%s].\", jobName, result.getStatus().getPhase(), task.getId());\n      long duration = System.currentTimeMillis() - start;\n      emitK8sPodMetrics(task, \"k8s/peon/startup/time\", duration);\n      return result;\n    });\n  }\n\n  /**\n   * Waits until a pod for the given job is created and ready to be monitored.\n   * <p>\n   * A pod can appear and dissapear in some cases, such as the task being canceled. In this case, null is returned and\n   * the caller should handle accordingly.\n   * </p>\n   *\n   * @param jobName  the name of the job whose pod we're waiting for\n   * @param howLong  the maximum time to wait\n   * @param timeUnit the time unit for the timeout","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java#L78-L114","documentation":"Thrown by KubernetesPeonClient.launchPeonJobAndWaitForStart when the Kubernetes Job for a task was submitted successfully and waitUntilPeonPodCreatedAndReady returned null, meaning the pod was expected to exist but could no longer be found after the wait window. Druid treats this as an internal state error because a freshly created job's pod should be present; disappearance usually means the job/pod was deleted, most commonly by task cancellation.","triggerScenarios":"createK8sJobWithRetries succeeds, then waitUntilPeonPodCreatedAndReady times out or observes the pod vanish (returns null) within the configured wait duration; the task's cancellation path or an external controller deleted the job/pod in the meantime.","commonSituations":"User or Overlord cancels a task while the peon pod is still starting; external cleanup scripts or namespace-wide eviction deleting young pods; very short peon start timeouts combined with slow pod scheduling; cluster autoscaler scaling away nodes before pods bind.","solutions":["Check whether the task was canceled (Overlord logs / task API); a cancellation racing startup is expected behavior","Increase the peon pod start timeout (e.g. druid.indexer.runner.k8s.peonTimeout / task start wait settings) if pods are slow to schedule","Investigate what removed the pod: job events via kubectl describe job, eviction notices, autoscaler or cleanup jobs","Re-run/resubmit the task; the condition is usually transient"],"exampleFix":"// no code fix; operational — check cancellation and pod events\nkubectl describe job <peon-job-name> -n <namespace>\nkubectl get events -n <namespace> | grep <pod-name>","handlingStrategy":"retry","validationCode":"// Before submission, ensure the task is not in a cancelled state\nTaskInfo info = overlordClient.getTaskInfo(taskId);\nif (info != null && info.getStatusCode() == TaskState.CANCELLED) {\n    return; // do not wait for a pod that will be deleted\n}","typeGuard":null,"tryCatchPattern":"try {\n    Pod pod = kubernetesPeonClient.launchPeonJobAndWaitForStart(job, task, timeout, unit);\n} catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"appeared and disappeared\")) {\n        // check if task was canceled; if so, treat as expected shutdown, else retry once\n        if (!overlordClient.getTaskInfo(task.getId()).isCancelled()) {\n            // resubmit or alert\n        }\n    } else {\n        throw e;\n    }\n}","preventionTips":["Increase peon pod startup timeout in slow-scheduling clusters","Avoid aggressive external cleanup jobs that delete young pods or jobs","Watch for task cancellations racing pod startup; they are the main benign cause"],"tags":["kubernetes","pod-lifecycle","druid-overlord","task-cancellation"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}