{"record":{"id":"cae3ee7e85f704c6","repo":"apache/druid","slug":"k8s-pod-with-label-job-name-s-not-found","errorCode":null,"errorMessage":"K8s pod with label[job-name=%s] not found","messagePattern":"K8s pod with label\\[job-name=(.+?)\\] not found","errorType":"exception","errorClass":"KubernetesResourceNotFoundException","httpStatus":404,"severity":"error","filePath":"extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java","lineNumber":473,"sourceCode":"   * @throws KubernetesResourceNotFoundException if the pod cannot be found after all retry attempts\n   * @throws DruidException if retrieval fails due to other errors\n   */\n  @VisibleForTesting\n  Pod getPeonPodWithRetries(KubernetesClient client, String jobName, int quietTries, int maxTries)\n  {\n    try {\n      return RetryUtils.retry(\n          () -> {\n            Optional<Pod> maybePod = getPeonPod(client, jobName);\n            if (maybePod.isPresent()) {\n              return maybePod.get();\n            }\n\n            // If the pod is missing, we can take a look at job events to discover potential problems with pod creation.\n            List<Event> events = getPeonEvents(client, jobName);\n\n            if (events.isEmpty()) {\n              throw new KubernetesResourceNotFoundException(\"K8s pod with label[job-name=%s] not found\", jobName);\n            } else {\n              Event latestEvent = events.get(events.size() - 1);\n              throw new KubernetesResourceNotFoundException(\n                  \"Job[%s] failed to create pods. Message[%s]\", jobName, latestEvent.getMessage());\n            }\n          },\n          this::shouldRetryWaitForStartingPeonPod, quietTries, maxTries\n      );\n    }\n    catch (KubernetesResourceNotFoundException e) {\n      throw e;\n    }\n    catch (Exception e) {\n      throw DruidException.defensive(e, \"Error when looking for K8s pod with label[job-name=%s]\", jobName);\n    }\n  }\n\n  /**","sourceCodeStart":455,"sourceCodeEnd":491,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java#L455-L491","documentation":"Thrown inside KubernetesPeonClient.getPeonPodWithRetries when retrying to find the peon pod with label job-name=<jobName> fails and, after inspecting the Kubernetes Job's events, no events exist at all. Druid throws KubernetesResourceNotFoundException because the pod simply does not exist and Kubernetes produced no event explaining why.","triggerScenarios":"The pod lookup by label selector consistently returns empty, getPeonEvents(client, jobName) returns an empty list (job created no events, or events expired / event TTL passed), and shouldRetryWaitForStartingPeonPod still allows the retry to conclude as a not-found error.","commonSituations":"The Job was deleted right after creation; the Kubernetes API server event TTL (default 1h) expired for old jobs; namespace mismatch (looking in a different namespace than where the job was created); the job object itself was never created due to earlier failures.","solutions":["Verify the job exists and inspect it directly: kubectl get job <jobName> -n <namespace> and kubectl describe job <jobName>","Check the Overlord is configured with the same namespace the tasks are launched in","Re-run the task if the job was externally deleted; Kubernetes events older than the TTL cannot be recovered","Ensure cluster RBAC allows reading pods, jobs and events in the namespace, so lookups and event inspection work"],"exampleFix":"// diagnostic, no code change\nkubectl get pods -l job-name=<jobName> -n <namespace>\nkubectl get job <jobName> -n <namespace> -o yaml","handlingStrategy":"retry","validationCode":"// Pre-check that the job exists before waiting for its pod\nJob job = client.batch().v1().jobs().inNamespace(namespace).withName(jobName).get();\nif (job == null) {\n    throw new IllegalStateException(\"Job \" + jobName + \" never created; check overlord logs and RBAC\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    Pod pod = client.getPeonPodWithRetries(jobName, ...);\n} catch (KubernetesResourceNotFoundException e) {\n    if (e.getMessage().startsWith(\"K8s pod with label\")) {\n        // pod gone and no events: likely deleted or events expired; resubmit task or inspect namespace\n    } else {\n        throw e;\n    }\n}","preventionTips":["Grant the Overlord RBAC read access to pods, jobs and events in the task namespace","Keep task namespace consistent between Overlord config and kubectl debugging","Do not delete jobs of tasks that are still starting"],"tags":["kubernetes","pod-not-found","druid-overlord","resource-lookup"],"backgroundTag":"resource-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}