Cache initialization for node role

Error message

Cache initialization for node role[%s] has already timed out. Ignoring cache initialization event.

What it means

This is a log warning from BaseNodeRoleWatcher.cacheInitialized. When the watcher's CountDownLatch-based cache initialization already timed out, any later cacheInitialized event for that node role is ignored because the watcher has already given up waiting and marked the initialization as timed out. It indicates a redundant late initialization signal, not a thrown exception.

Solutions

Increase druid.discovery.cache.timeoutMillis so the cache normally initializes before the timeout.
Investigate why service discovery (ZooKeeper/Announcer) is slow: check ZK latency, GC pauses, network.
Restart the service after the environment issue is fixed so the cache initializes within the window.
Verify full node information is present; if the timeout already fired the service may lack info about peer nodes until restart.

Example fix

// before
druid.discovery.cache.timeoutMillis=PT10S
// after
druid.discovery.cache.timeoutMillis=PT60S

Defensive patterns

Strategy: validation

Validate before calling

// Ensure cache timeout exceeds worst-case discovery latency
duration timeout = config.getDiscoveryCacheTimeout();
if (timeout.isShorterThan(Duration.standardSeconds(30))) {
  log.warn("discovery cache timeout %s may be too short for large clusters", timeout);
}

Prevention

Size druid.discovery.cache.timeoutMillis above worst-case ZK/curator cache warm-up time
Monitor service-discovery initialization duration at startup
Alert on this warning; a timed-out cache means partial peer information until restart

When it happens

Trigger: cacheInitialized(timedOut=false) is invoked after the timeout runnable already called cacheInitialized(timedOut=true) for the same role; e.g. the service-discovery cache finally populated just after the druid.discovery.cache.timeoutMillis window elapsed.

Common situations: Slow or overloaded ZooKeeper/coordinator causing listener population to exceed the cache initialization timeout; services starting under heavy load or with network latency slightly above the configured timeout.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/539b99e923f3b524. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/discovery/BaseNodeRoleWatcher.java:251

        safeSchedule(
            () -> listener.nodesRemoved(nodeRemoved),
            "Exception occurred in nodeRemoved(node [%s] of role [%s]) in listener [%s].",
            druidNode.getDruidNode().getUriToUse(),
            druidNode.getNodeRole().getJsonName(),
            listener
        );
      }
    }
  }

  public void cacheInitialized()
  {
    synchronized (lock) {
      // No need to wait on CountDownLatch, because we are holding the lock under which it could only be
      // counted down.
      if (cacheInitialized.getCount() == 0) {
        if (cacheInitializationTimedOut) {
          LOGGER.warn(
              "Cache initialization for node role[%s] has already timed out. Ignoring cache initialization event.",
              nodeRole.getJsonName()
          );
        } else {
          LOGGER.error(
              "Cache for node role[%s] is already initialized. ignoring cache initialization event.",
              nodeRole.getJsonName()
          );
        }
        return;
      }

      cacheInitialized(false);
    }
  }

  private void cacheInitializedTimedOut()
  {

View on GitHub (pinned to 9b90983fd2)