{"record":{"id":"35cd7a17d4eac292","repo":"apache/druid","slug":"expection-in-listing-pods-code-d-and-error-s","errorCode":null,"errorMessage":"Expection in listing pods, code[%d] and error[%s].","messagePattern":"Expection in listing pods, code\\[(.+?)\\] and error\\[(.+?)\\]\\.","errorType":"exception","errorClass":"RuntimeException (Druid RE)","httpStatus":null,"severity":"error","filePath":"extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java","lineNumber":133,"sourceCode":"      Preconditions.checkState(podList != null, \"WTH: NULL podList\");\n\n      Map<String, DiscoveryDruidNode> allNodes = new HashMap();\n      for (V1Pod podDef : podList.getItems()) {\n        if (!isPodReady(podDef)) {\n          LOGGER.info(\n              \"Ignoring pod[%s] for role[%s] during list: pod has discovery label but is not yet reporting as ready.\",\n              podDef.getMetadata().getName(),\n              nodeRole\n          );\n          continue;\n        }\n        DiscoveryDruidNode node = getDiscoveryDruidNodeFromPodDef(nodeRole, podDef);\n        allNodes.put(node.getDruidNode().getHostAndPortToUse(), node);\n      }\n      return new DiscoveryDruidNodeList(podList.getMetadata().getResourceVersion(), allNodes);\n    }\n    catch (ApiException ex) {\n      throw new RE(ex, \"Expection in listing pods, code[%d] and error[%s].\", ex.getCode(), ex.getResponseBody());\n    }\n  }\n\n  /**\n   * Check whether a pod's containers are all running and ready. This is used to filter out pods\n   * whose containers have been OOM-killed or are otherwise not serving traffic, even though the\n   * pod itself still exists and retains its Druid announcement labels.\n   */\n  static boolean isPodReady(V1Pod pod)\n  {\n    if (pod.getStatus() == null) {\n      return false;\n    }\n    List<V1ContainerStatus> containerStatuses = pod.getStatus().getContainerStatuses();\n    if (containerStatuses == null || containerStatuses.isEmpty()) {\n      return false;\n    }\n    return containerStatuses.stream().allMatch(cs -> Boolean.TRUE.equals(cs.getReady()));","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java#L115-L151","documentation":"DefaultK8sApiClient.listPods lists pods matching the Druid node label selectors and builds a DiscoveryDruidNodeList. An ApiException from the Kubernetes client is rethrown as a Druid RE reading 'Expection in listing pods, code[%d] and error[%s].' (note the typo 'Expection' upstream) with the HTTP code and response body.","triggerScenarios":"listPods(podNamespace, taskLabelValue, nodeRole) where the core_v1Api.listNamespacedPod call throws ApiException: 403 RBAC denial, 404 bad namespace, 401 auth failure, 429 throttling, or connection failure to the API server.","commonSituations":"Service account lacking 'pods/list' permission in the namespace; wrong druid namespace config; API server briefly unavailable; label-selector/namespace typos; client kubeconfig not mounted in the pod.","solutions":["Check the code in the RE message: 403 -> grant RBAC get/list/watch on pods in the namespace; 404 -> fix namespace config","Verify service-account token/kubeconfig is present and valid inside the pod","Retry on transient codes (429, 5xx, connection errors) — callers like K8sDiscoveryClient usually retry","Confirm the label selector and namespace match where Druid pods actually run"],"exampleFix":"// before\nDiscoveryDruidNodeList pods = k8sClient.listPods(ns, label, role);\n// after\nDiscoveryDruidNodeList pods;\ntry {\n  pods = k8sClient.listPods(ns, label, role);\n} catch (RE e) {\n  LOG.warn(e, \"listing pods failed; falling back to cached node list\");\n  pods = cachedNodeList;\n}","handlingStrategy":"retry","validationCode":"// precheck RBAC and namespace\ncoreV1Api.listNamespacedPod(ns).labelSelector(selector).execute(); // throws ApiException if unauthorized","typeGuard":"static boolean isTransientK8sError(ApiException ex) {\n  return ex.getCode() == 429 || ex.getCode() >= 500;\n}","tryCatchPattern":"try {\n  nodes = client.listPods(ns, label, role);\n} catch (RE e) {\n  if (e.getMessage().contains(\"code[403]\")) throw new IllegalStateException(\"RBAC: grant pods/list\", e);\n  if (e.getMessage().matches(\".*code\\\\[(429|5\\\\d\\\\d)\\\\].*\")) { /* retry with backoff */ }\n  else throw e;\n}","preventionTips":["Bind a Role allowing get/list/watch on pods in the target namespace","Verify namespace and label-selector config match where Druid pods actually run","Retry transient API-server errors; K8sDiscoveryClient already retries — keep that enabled","Ensure the service-account token volume is mounted in the calling pod"],"tags":["kubernetes","pod-list","api-client"],"backgroundTag":"api-error-response","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}