apache/druid · warning · RuntimeException (Druid RE)

Exception while closing watch.

Error message

Exception while closing watch.

What it means

Wraps an IOException from the Kubernetes client's Watch.close() when tearing down a pod watch stream. The watch's underlying HTTP stream could not be cleanly closed; the error is rethrown as a Druid RE so callers of the WatchResult know cleanup failed.

Source

Thrown at extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java:298

          }

          return false;
        }

        @Override
        public Watch.Response<DiscoveryDruidNodeAndResourceVersion> next()
        {
          return obj;
        }

        @Override
        public void close()
        {
          try {
            watch.close();
          }
          catch (IOException ex) {
            throw new RE(ex, "Exception while closing watch.");
          }
        }
      };
    }
    catch (ApiException ex) {
      if (ex.getCode() == 410) {
        // k8s no longer has history that we need
        return null;
      }

      throw new RE(ex, "Expection in watching pods, code[%d] and error[%s].", ex.getCode(), ex.getResponseBody());
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Log and continue if the watch is being discarded anyway — the exception only affects cleanup, not data.
2. Ensure watchPods results are closed promptly; long-lived broken streams are more likely to fail on close.
3. Check connectivity to the K8s API server and client http timeout settings.
4. Retry the whole watchPods cycle; a fresh watch usually closes fine.

Example fix

// before
try (WatchResult wr = client.watchPods(ns, selector, rv, role)) { ... }
// after: tolerate close failures
WatchResult wr = client.watchPods(ns, selector, rv, role);
try { ... } finally { try { wr.close(); } catch (Exception e) { LOG.warn(e, "watch close failed"); } }

try (WatchResult wr = apiClient.watchPods(ns, selector, rv, role)) {
  // consume events
} catch (Exception e) {
  LOG.warn(e, "watch close/consume failed; resyncing");
}

When it happens

Trigger: Closing a WatchResult obtained from watchPods when the underlying okhttp/http client throws IOException during shutdown — e.g. the connection was already broken, the API server dropped the stream, or client resources were already released.

Common situations: K8s API server or network connection terminated before close; watch already expired (410 Gone previously) and the stale stream errors on close; abrupt pod restarts during watch teardown.

Related errors

• Expection in watching pods, code[%d] and error[%s].
• Failed to remove output directory [%s] for segment pulled fr
• IOException wrapping underlying cause
• Thread interrupted. Couldn't delete all tasklogs.
• Failed to patch pod[%s/%s], code[%d], error[%s].

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/67e48eba8580f748. Report an issue: GitHub.