{"record":{"id":"846e87f0f1aefb7b","repo":"apache/druid","slug":"job-s-failed-to-create-pods-message-s","errorCode":null,"errorMessage":"Job[%s] failed to create pods. Message[%s]","messagePattern":"Job\\[(.+?)\\] failed to create pods\\. Message\\[(.+?)\\]","errorType":"exception","errorClass":"KubernetesResourceNotFoundException","httpStatus":404,"severity":"error","filePath":"extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java","lineNumber":476,"sourceCode":"  @VisibleForTesting\n  Pod getPeonPodWithRetries(KubernetesClient client, String jobName, int quietTries, int maxTries)\n  {\n    try {\n      return RetryUtils.retry(\n          () -> {\n            Optional<Pod> maybePod = getPeonPod(client, jobName);\n            if (maybePod.isPresent()) {\n              return maybePod.get();\n            }\n\n            // If the pod is missing, we can take a look at job events to discover potential problems with pod creation.\n            List<Event> events = getPeonEvents(client, jobName);\n\n            if (events.isEmpty()) {\n              throw new KubernetesResourceNotFoundException(\"K8s pod with label[job-name=%s] not found\", jobName);\n            } else {\n              Event latestEvent = events.get(events.size() - 1);\n              throw new KubernetesResourceNotFoundException(\n                  \"Job[%s] failed to create pods. Message[%s]\", jobName, latestEvent.getMessage());\n            }\n          },\n          this::shouldRetryWaitForStartingPeonPod, quietTries, maxTries\n      );\n    }\n    catch (KubernetesResourceNotFoundException e) {\n      throw e;\n    }\n    catch (Exception e) {\n      throw DruidException.defensive(e, \"Error when looking for K8s pod with label[job-name=%s]\", jobName);\n    }\n  }\n\n  /**\n   * Determines if this exception, specifically when containing Kubernetes job event messages, permits a retry attempt.\n   * <p>\n   * The method checks the exception message against a predefined list of Kubernetes event messages.","sourceCodeStart":458,"sourceCodeEnd":494,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java#L458-L494","documentation":"Thrown inside KubernetesPeonClient.getPeonPodWithRetries when the peon pod with label job-name=<jobName> is missing but Kubernetes Job events DO exist; Druid surfaces the latest event message inside a KubernetesResourceNotFoundException so the operator can see why the Job failed to create its pod (e.g. quota exceeded, image pull error, scheduling failure).","triggerScenarios":"Pod lookup by job-name label returns empty across retries; getPeonEvents returns at least one event; the last event's message (e.g. 'Exceeded resource quota', 'FailedScheduling', 'Forbidden') is embedded into the thrown exception.","commonSituations":"Namespace ResourceQuota blocking pod creation; nodes unschedulable (FailedScheduling) with pending pods; RBAC or admission webhooks rejecting pod creation; image pull policies failing before pod creation completes.","solutions":["Read the event message in the exception and address its cause (e.g. raise ResourceQuota, fix scheduling constraints, fix image reference)","Run kubectl describe job <jobName> -n <namespace> to see all recent events","Check pod creation RBAC for the Overlord service account if the message is Forbidden","Reduce pod resource requests or add schedulable nodes if FailedScheduling persists"],"exampleFix":"// diagnostic, no code change\nkubectl describe job <jobName> -n <namespace>   # shows the same event message\nkubectl get resourcequota -n <namespace>","handlingStrategy":"try-catch","validationCode":"// Validate namespace quota headroom before launching tasks\nResourceQuota quota = client.resourceQuotas().inNamespace(namespace).withName(\"compute-resources\").get();\nif (quota != null && quota.getStatus() != null) {\n    // compare used vs hard for pods/cpu/memory and alert when near limits\n}","typeGuard":null,"tryCatchPattern":"try {\n    Pod pod = client.getPeonPodWithRetries(jobName, ...);\n} catch (KubernetesResourceNotFoundException e) {\n    // message contains \"failed to create pods. Message[<latest K8s event>]\"\n    String k8sEvent = extractEventMessage(e.getMessage());\n    // branch on FailedScheduling / quota / Forbidden and fix infrastructure accordingly\n}","preventionTips":["Monitor namespace ResourceQuota usage where peon pods are created","Ensure schedulable nodes exist for task pod resource requests","Check admission webhooks/RBAC allow pod creation by the Overlord service account"],"tags":["kubernetes","pod-not-found","scheduling","druid-overlord"],"backgroundTag":"resource-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}