apache/druid · error · RuntimeException (Druid RE)

Expection in listing pods, code[%d] and error[%s].

Error message

Expection in listing pods, code[%d] and error[%s].

What it means

DefaultK8sApiClient.listPods lists pods matching the Druid node label selectors and builds a DiscoveryDruidNodeList. An ApiException from the Kubernetes client is rethrown as a Druid RE reading 'Expection in listing pods, code[%d] and error[%s].' (note the typo 'Expection' upstream) with the HTTP code and response body.

Source

Thrown at extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java:133

      Preconditions.checkState(podList != null, "WTH: NULL podList");

      Map<String, DiscoveryDruidNode> allNodes = new HashMap();
      for (V1Pod podDef : podList.getItems()) {
        if (!isPodReady(podDef)) {
          LOGGER.info(
              "Ignoring pod[%s] for role[%s] during list: pod has discovery label but is not yet reporting as ready.",
              podDef.getMetadata().getName(),
              nodeRole
          );
          continue;
        }
        DiscoveryDruidNode node = getDiscoveryDruidNodeFromPodDef(nodeRole, podDef);
        allNodes.put(node.getDruidNode().getHostAndPortToUse(), node);
      }
      return new DiscoveryDruidNodeList(podList.getMetadata().getResourceVersion(), allNodes);
    }
    catch (ApiException ex) {
      throw new RE(ex, "Expection in listing pods, code[%d] and error[%s].", ex.getCode(), ex.getResponseBody());
    }
  }

  /**
   * Check whether a pod's containers are all running and ready. This is used to filter out pods
   * whose containers have been OOM-killed or are otherwise not serving traffic, even though the
   * pod itself still exists and retains its Druid announcement labels.
   */
  static boolean isPodReady(V1Pod pod)
  {
    if (pod.getStatus() == null) {
      return false;
    }
    List<V1ContainerStatus> containerStatuses = pod.getStatus().getContainerStatuses();
    if (containerStatuses == null || containerStatuses.isEmpty()) {
      return false;
    }
    return containerStatuses.stream().allMatch(cs -> Boolean.TRUE.equals(cs.getReady()));

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Check the code in the RE message: 403 -> grant RBAC get/list/watch on pods in the namespace; 404 -> fix namespace config
2. Verify service-account token/kubeconfig is present and valid inside the pod
3. Retry on transient codes (429, 5xx, connection errors) — callers like K8sDiscoveryClient usually retry
4. Confirm the label selector and namespace match where Druid pods actually run

Example fix

// before
DiscoveryDruidNodeList pods = k8sClient.listPods(ns, label, role);
// after
DiscoveryDruidNodeList pods;
try {
  pods = k8sClient.listPods(ns, label, role);
} catch (RE e) {
  LOG.warn(e, "listing pods failed; falling back to cached node list");
  pods = cachedNodeList;
}

// precheck RBAC and namespace
coreV1Api.listNamespacedPod(ns).labelSelector(selector).execute(); // throws ApiException if unauthorized

static boolean isTransientK8sError(ApiException ex) {
  return ex.getCode() == 429 || ex.getCode() >= 500;
}

try {
  nodes = client.listPods(ns, label, role);
} catch (RE e) {
  if (e.getMessage().contains("code[403]")) throw new IllegalStateException("RBAC: grant pods/list", e);
  if (e.getMessage().matches(".*code\\[(429|5\\d\\d)\\].*")) { /* retry with backoff */ }
  else throw e;
}

When it happens

Trigger: listPods(podNamespace, taskLabelValue, nodeRole) where the core_v1Api.listNamespacedPod call throws ApiException: 403 RBAC denial, 404 bad namespace, 401 auth failure, 429 throttling, or connection failure to the API server.

Common situations: Service account lacking 'pods/list' permission in the namespace; wrong druid namespace config; API server briefly unavailable; label-selector/namespace typos; client kubeconfig not mounted in the pod.

Related errors

• Failed to patch pod[%s/%s], code[%d], error[%s].
• Failed to deserialize DiscoveryDruidNode[%s]
• Exception while closing watch.
• Expection in watching pods, code[%d] and error[%s].
• Failed to get current leader for [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/35cd7a17d4eac292. Report an issue: GitHub.