What causes "K8s pod with label[job-name=%s] not found" in apache/druid?

Verify the job exists and inspect it directly: kubectl get job -n and kubectl describe job

apache/druid · error · KubernetesResourceNotFoundException

K8s pod with label[job-name=%s] not found

Error message

K8s pod with label[job-name=%s] not found

What it means

Thrown inside KubernetesPeonClient.getPeonPodWithRetries when retrying to find the peon pod with label job-name=<jobName> fails and, after inspecting the Kubernetes Job's events, no events exist at all. Druid throws KubernetesResourceNotFoundException because the pod simply does not exist and Kubernetes produced no event explaining why.

Source

Thrown at extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/KubernetesPeonClient.java:473

   * @throws KubernetesResourceNotFoundException if the pod cannot be found after all retry attempts
   * @throws DruidException if retrieval fails due to other errors
   */
  @VisibleForTesting
  Pod getPeonPodWithRetries(KubernetesClient client, String jobName, int quietTries, int maxTries)
  {
    try {
      return RetryUtils.retry(
          () -> {
            Optional<Pod> maybePod = getPeonPod(client, jobName);
            if (maybePod.isPresent()) {
              return maybePod.get();
            }

            // If the pod is missing, we can take a look at job events to discover potential problems with pod creation.
            List<Event> events = getPeonEvents(client, jobName);

            if (events.isEmpty()) {
              throw new KubernetesResourceNotFoundException("K8s pod with label[job-name=%s] not found", jobName);
            } else {
              Event latestEvent = events.get(events.size() - 1);
              throw new KubernetesResourceNotFoundException(
                  "Job[%s] failed to create pods. Message[%s]", jobName, latestEvent.getMessage());
            }
          },
          this::shouldRetryWaitForStartingPeonPod, quietTries, maxTries
      );
    }
    catch (KubernetesResourceNotFoundException e) {
      throw e;
    }
    catch (Exception e) {
      throw DruidException.defensive(e, "Error when looking for K8s pod with label[job-name=%s]", jobName);
    }
  }

  /**

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the job exists and inspect it directly: kubectl get job <jobName> -n <namespace> and kubectl describe job <jobName>
Check the Overlord is configured with the same namespace the tasks are launched in
Re-run the task if the job was externally deleted; Kubernetes events older than the TTL cannot be recovered
Ensure cluster RBAC allows reading pods, jobs and events in the namespace, so lookups and event inspection work

Example fix

// diagnostic, no code change
kubectl get pods -l job-name=<jobName> -n <namespace>
kubectl get job <jobName> -n <namespace> -o yaml

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check that the job exists before waiting for its pod
Job job = client.batch().v1().jobs().inNamespace(namespace).withName(jobName).get();
if (job == null) {
    throw new IllegalStateException("Job " + jobName + " never created; check overlord logs and RBAC");
}

Try / catch

try {
    Pod pod = client.getPeonPodWithRetries(jobName, ...);
} catch (KubernetesResourceNotFoundException e) {
    if (e.getMessage().startsWith("K8s pod with label")) {
        // pod gone and no events: likely deleted or events expired; resubmit task or inspect namespace
    } else {
        throw e;
    }
}

Prevention

Grant the Overlord RBAC read access to pods, jobs and events in the task namespace
Keep task namespace consistent between Overlord config and kubectl debugging
Do not delete jobs of tasks that are still starting

When it happens

Trigger: The pod lookup by label selector consistently returns empty, getPeonEvents(client, jobName) returns an empty list (job created no events, or events expired / event TTL passed), and shouldRetryWaitForStartingPeonPod still allows the retry to conclude as a not-found error.

Common situations: The Job was deleted right after creation; the Kubernetes API server event TTL (default 1h) expired for old jobs; namespace mismatch (looking in a different namespace than where the job was created); the job object itself was never created due to earlier failures.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/cae3ee7e85f704c6. Report an issue: GitHub.