apache/druid · error · RuntimeException (Druid RE)

Expection in watching pods, code[%d] and error[%s].

Error message

Expection in watching pods, code[%d] and error[%s].

What it means

Wraps a Kubernetes ApiException from the watch-pods call when the failure is not 410 Gone. The message includes the API response code and raw response body. 410 is deliberately handled as 'resource version history gone' and returns null so the caller can resync; any other failure (401/403/5xx, connection issues) raises this RE.

Source

Thrown at extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java:309

        @Override
        public void close()
        {
          try {
            watch.close();
          }
          catch (IOException ex) {
            throw new RE(ex, "Exception while closing watch.");
          }
        }
      };
    }
    catch (ApiException ex) {
      if (ex.getCode() == 410) {
        // k8s no longer has history that we need
        return null;
      }

      throw new RE(ex, "Expection in watching pods, code[%d] and error[%s].", ex.getCode(), ex.getResponseBody());
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Read code and error body in the message: 403 → fix RBAC role (get/list/watch pods in the namespace); 401 → fix service-account token.
2. Confirm the namespace and labelSelector in druid.k8s.* config point at real Druid pods.
3. Check K8s API server health (kubectl get pods) and network path from the Druid process.
4. After fixing, restart/retry; discovery resyncs from a fresh watch.

// pre-flight: ensure RBAC and namespace before watching
boolean canWatch = authCanI("list", "pods", namespace); // kubectl auth can-i list pods -n <ns>
if (!canWatch) throw new IllegalStateException("Service account cannot watch pods in " + namespace);

try {
  WatchResult wr = client.watchPods(ns, selector, rv, role);
  if (wr == null) { /* 410: resync from list */ }
} catch (RE e) {
  Throwable c = e.getCause();
  if (c instanceof ApiException && ((ApiException) c).getCode() == 403) { fixRbac(); }
  else { backoffAndRetry(); }
}

When it happens

Trigger: DefaultK8sApiClient.watchPods receives a non-410 ApiException from the Kubernetes API while establishing or streaming the pod watch — e.g. expired token, RBAC denial, API server 500, or connection refused.

Common situations: Service account lacking list/watch RBAC on pods (403); expired or misconfigured kube credentials (401); K8s API server outage (5xx); wrong namespace or label selector.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Failed to get current leader for [%s]
• Failed to announce DiscoveryDruidNode[%s]
• Exception while closing watch.
• Exception while watching for role[%s], will retry.
• Failed to fetch Avro schema id[%s] from registry. Check if t

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/e8c4a20a8506bded. Report an issue: GitHub.