apache/druid · warning

Cache for node role [ ] could not be initialized before…

Error message

Cache for node role [%s] could not be initialized before timeout. This service may not have full information about other nodes of type [%s].

What it means

Log warning from BaseNodeRoleWatcher.cacheInitialized(timedOut=true), fired when the service-discovery cache for a node role was not populated within the configured timeout. The service marks cacheInitializationTimedOut and continues operating, possibly without complete information about other nodes of the same role. Late initialization events are then ignored.

Solutions

1. Check ZooKeeper connectivity and curator session state in the logs at startup.
2. Raise druid.discovery.cache.timeoutMillis (e.g. from PT10S to PT1M) to accommodate slow clusters.
3. Confirm nodes of the role are actually running and announcing their services via the coordinator/announcer.
4. Restart the service once discovery is healthy to re-attempt cache initialization.

Example fix

// before
druid.discovery.cache.timeoutMillis=PT10S
// after
druid.discovery.cache.timeoutMillis=PT1M

// Verify ZK reachability before/at startup and that the cache timeout is adequate
if (!zkHealthCheck.await(30, SECONDS)) {
  throw new IllegalStateException("ZooKeeper not reachable; discovery cache will time out");
}

When it happens

Trigger: The CountDownLatch for cache population is not counted down before the scheduled timeout task runs cacheInitialized(true); e.g. no nodes of the role announced, or the watcher's addListener/child watchers have not delivered updates within druid.discovery.cache.timeoutMillis.

Common situations: ZooKeeper connectivity problems or session instability at startup; configured timeout too small for cluster size; a role with zero instances so the 'at least one node' expectation never completes; heavy startup load delaying curator cache population.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Cache initialization for node role
• Node[ ] disappeared but was unknown for service listener […
• Node[ ] discovered but already exists [ ].
• Node [ ] of role [ ] went offline.
• Watcher for node role

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/3a1219329733ee1a. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/discovery/BaseNodeRoleWatcher.java:284

  }

  private void cacheInitializedTimedOut()
  {
    synchronized (lock) {
      // No need to wait on CountDownLatch, because we are holding the lock under which it could only be
      // counted down.
      if (cacheInitialized.getCount() != 0) {
        cacheInitialized(true);
      }
    }
  }

  // This method is called only once with either timedOut = true or false, but not both.
  @GuardedBy("lock")
  private void cacheInitialized(boolean timedOut)
  {
    if (timedOut) {
      LOGGER.warn(
          "Cache for node role [%s] could not be initialized before timeout. "
          + "This service may not have full information about other nodes of type [%s].",
          nodeRole.getJsonName(),
          nodeRole.getJsonName()
      );
      cacheInitializationTimedOut = true;
    }

    // It is important to take a snapshot here as list of nodes might change by the time listeners process
    // the changes.
    List<DiscoveryDruidNode> currNodes = Lists.newArrayList(nodes.values());
    LOGGER.info(
        "Node watcher of role [%s] is now initialized with %d nodes.",
        nodeRole.getJsonName(),
        currNodes.size());

    for (DruidNodeDiscovery.Listener listener : nodeListeners) {
      safeSchedule(

View on GitHub (pinned to 9b90983fd2)