apache/druid · error · KubernetesResourceNotFoundException

Job[%s] failed to create pods. Message[%s]

Error message

Job[%s] failed to create pods. Message[%s]

What it means

Thrown inside KubernetesPeonClient.getPeonPodWithRetries when the peon pod with label job-name=<jobName> is missing but Kubernetes Job events DO exist; Druid surfaces the latest event message inside a KubernetesResourceNotFoundException so the operator can see why the Job failed to create its pod (e.g. quota exceeded, image pull error, scheduling failure).

Source

Thrown at extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java:476

  @VisibleForTesting
  Pod getPeonPodWithRetries(KubernetesClient client, String jobName, int quietTries, int maxTries)
  {
    try {
      return RetryUtils.retry(
          () -> {
            Optional<Pod> maybePod = getPeonPod(client, jobName);
            if (maybePod.isPresent()) {
              return maybePod.get();
            }

            // If the pod is missing, we can take a look at job events to discover potential problems with pod creation.
            List<Event> events = getPeonEvents(client, jobName);

            if (events.isEmpty()) {
              throw new KubernetesResourceNotFoundException("K8s pod with label[job-name=%s] not found", jobName);
            } else {
              Event latestEvent = events.get(events.size() - 1);
              throw new KubernetesResourceNotFoundException(
                  "Job[%s] failed to create pods. Message[%s]", jobName, latestEvent.getMessage());
            }
          },
          this::shouldRetryWaitForStartingPeonPod, quietTries, maxTries
      );
    }
    catch (KubernetesResourceNotFoundException e) {
      throw e;
    }
    catch (Exception e) {
      throw DruidException.defensive(e, "Error when looking for K8s pod with label[job-name=%s]", jobName);
    }
  }

  /**
   * Determines if this exception, specifically when containing Kubernetes job event messages, permits a retry attempt.
   * <p>
   * The method checks the exception message against a predefined list of Kubernetes event messages.

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Read the event message in the exception and address its cause (e.g. raise ResourceQuota, fix scheduling constraints, fix image reference)
2. Run kubectl describe job <jobName> -n <namespace> to see all recent events
3. Check pod creation RBAC for the Overlord service account if the message is Forbidden
4. Reduce pod resource requests or add schedulable nodes if FailedScheduling persists

Example fix

// diagnostic, no code change
kubectl describe job <jobName> -n <namespace>   # shows the same event message
kubectl get resourcequota -n <namespace>

// Validate namespace quota headroom before launching tasks
ResourceQuota quota = client.resourceQuotas().inNamespace(namespace).withName("compute-resources").get();
if (quota != null && quota.getStatus() != null) {
    // compare used vs hard for pods/cpu/memory and alert when near limits
}

try {
    Pod pod = client.getPeonPodWithRetries(jobName, ...);
} catch (KubernetesResourceNotFoundException e) {
    // message contains "failed to create pods. Message[<latest K8s event>]"
    String k8sEvent = extractEventMessage(e.getMessage());
    // branch on FailedScheduling / quota / Forbidden and fix infrastructure accordingly
}

When it happens

Trigger: Pod lookup by job-name label returns empty across retries; getPeonEvents returns at least one event; the last event's message (e.g. 'Exceeded resource quota', 'FailedScheduling', 'Forbidden') is embedded into the thrown exception.

Common situations: Namespace ResourceQuota blocking pod creation; nodes unschedulable (FailedScheduling) with pending pods; RBAC or admission webhooks rejecting pod creation; image pull policies failing before pod creation completes.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• K8s pod with label[job-name=%s] not found
• Invalid pod adapter [%s], only pod adapter [%s] can be speci
• Pod adapter [%s] requires the local Overlord pod namespace f
• K8s pod for the task[%s] appeared and disappeared. It can ha
• Pod template task adapter requires a base pod template to be

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/846e87f0f1aefb7b. Report an issue: GitHub.