{"record":{"id":"539b99e923f3b524","repo":"apache/druid","slug":"cache-initialization-for-node-role-s-has-already","errorCode":null,"errorMessage":"Cache initialization for node role[%s] has already timed out. Ignoring cache initialization event.","messagePattern":"Cache initialization for node role\\[(.+?)\\] has already timed out\\. Ignoring cache initialization event\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"server/src/main/java/org/apache/druid/discovery/BaseNodeRoleWatcher.java","lineNumber":251,"sourceCode":"        safeSchedule(\n            () -> listener.nodesRemoved(nodeRemoved),\n            \"Exception occurred in nodeRemoved(node [%s] of role [%s]) in listener [%s].\",\n            druidNode.getDruidNode().getUriToUse(),\n            druidNode.getNodeRole().getJsonName(),\n            listener\n        );\n      }\n    }\n  }\n\n  public void cacheInitialized()\n  {\n    synchronized (lock) {\n      // No need to wait on CountDownLatch, because we are holding the lock under which it could only be\n      // counted down.\n      if (cacheInitialized.getCount() == 0) {\n        if (cacheInitializationTimedOut) {\n          LOGGER.warn(\n              \"Cache initialization for node role[%s] has already timed out. Ignoring cache initialization event.\",\n              nodeRole.getJsonName()\n          );\n        } else {\n          LOGGER.error(\n              \"Cache for node role[%s] is already initialized. ignoring cache initialization event.\",\n              nodeRole.getJsonName()\n          );\n        }\n        return;\n      }\n\n      cacheInitialized(false);\n    }\n  }\n\n  private void cacheInitializedTimedOut()\n  {","sourceCodeStart":233,"sourceCodeEnd":269,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/discovery/BaseNodeRoleWatcher.java#L233-L269","documentation":"This is a log warning from BaseNodeRoleWatcher.cacheInitialized. When the watcher's CountDownLatch-based cache initialization already timed out, any later cacheInitialized event for that node role is ignored because the watcher has already given up waiting and marked the initialization as timed out. It indicates a redundant late initialization signal, not a thrown exception.","triggerScenarios":"cacheInitialized(timedOut=false) is invoked after the timeout runnable already called cacheInitialized(timedOut=true) for the same role; e.g. the service-discovery cache finally populated just after the druid.discovery.cache.timeoutMillis window elapsed.","commonSituations":"Slow or overloaded ZooKeeper/coordinator causing listener population to exceed the cache initialization timeout; services starting under heavy load or with network latency slightly above the configured timeout.","solutions":["Increase druid.discovery.cache.timeoutMillis so the cache normally initializes before the timeout.","Investigate why service discovery (ZooKeeper/Announcer) is slow: check ZK latency, GC pauses, network.","Restart the service after the environment issue is fixed so the cache initializes within the window.","Verify full node information is present; if the timeout already fired the service may lack info about peer nodes until restart."],"exampleFix":"// before\ndruid.discovery.cache.timeoutMillis=PT10S\n// after\ndruid.discovery.cache.timeoutMillis=PT60S","handlingStrategy":"validation","validationCode":"// Ensure cache timeout exceeds worst-case discovery latency\nduration timeout = config.getDiscoveryCacheTimeout();\nif (timeout.isShorterThan(Duration.standardSeconds(30))) {\n  log.warn(\"discovery cache timeout %s may be too short for large clusters\", timeout);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size druid.discovery.cache.timeoutMillis above worst-case ZK/curator cache warm-up time","Monitor service-discovery initialization duration at startup","Alert on this warning; a timed-out cache means partial peer information until restart"],"tags":["service-discovery","timeout","zookeeper"],"backgroundTag":"request-timeout","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}