{"record":{"id":"e3e6412607fd08d9","repo":"apache/druid","slug":"node-s-of-role-s-went-offline","errorCode":null,"errorMessage":"Node [%s] of role [%s] went offline.","messagePattern":"Node \\[(.+?)\\] of role \\[(.+?)\\] went offline\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"info","filePath":"server/src/main/java/org/apache/druid/discovery/BaseNodeRoleWatcher.java","lineNumber":209,"sourceCode":"        LOGGER.error(\n            \"Node [%s] of role [%s] removal ignored due to mismatched role (expected role [%s]).\",\n            druidNode.getDruidNode().getUriToUse(),\n            druidNode.getNodeRole().getJsonName(),\n            nodeRole.getJsonName()\n        );\n        return;\n      }\n\n      if (skipIfUnknown && !nodes.containsKey(druidNode.getDruidNode().getHostAndPortToUse())) {\n        LOGGER.debug(\n            \"Ignoring removal of node [%s] of role [%s] because it is not known to be present in the cache.\",\n            druidNode.getDruidNode().getUriToUse(),\n            nodeRole.getJsonName()\n        );\n        return;\n      }\n\n      LOGGER.warn(\"Node [%s] of role [%s] went offline.\", druidNode.getDruidNode().getUriToUse(), nodeRole.getJsonName());\n\n      removeNode(druidNode);\n    }\n  }\n\n  @GuardedBy(\"lock\")\n  private void removeNode(DiscoveryDruidNode druidNode)\n  {\n    DiscoveryDruidNode prev = nodes.remove(druidNode.getDruidNode().getHostAndPortToUse());\n\n    if (prev == null) {\n      LOGGER.error(\n          \"Noticed disappearance of unknown druid node [%s] of role [%s].\",\n          druidNode.getDruidNode().getUriToUse(),\n          druidNode.getNodeRole().getJsonName()\n      );\n      return;\n    }","sourceCodeStart":191,"sourceCodeEnd":227,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/discovery/BaseNodeRoleWatcher.java#L191-L227","documentation":"BaseNodeRoleWatcher.childRemoved() detected that an announced child znode for a known Druid node was deleted in ZK, so the node is logged as going offline and removed from the in-memory node map (removeNode), which then notifies registered listeners. This is the standard notification that a peer service instance left the cluster (shutdown or session loss).","triggerScenarios":"A ZK CHILD_REMOVED event for a node present in the watcher's map: the peer process stopped gracefully (closes its ephemeral announcement), its ZK session expired after a crash, or ZK cleaned up ephemerals on disconnect.","commonSituations":"Normal service restarts/scaling-down of middle managers, brokers, or historicals; crash-induced ZK session expiry; network partitions causing session loss; load balancer or ops tooling deleting znodes.","solutions":["No action if the peer was intentionally stopped — this is the expected offline notification.","If unexpected, check the offline node's logs/host for crash, OOM kill, or loss of ZK connectivity.","Review druid.curator.sessionTimeout and network stability if nodes flapping on/off causes churn.","Ensure dependent services (e.g. SQL planner, task runners) handle the node-gone listener callback gracefully."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// nodeGone handlers must tolerate removal of nodes they never used\n@Override\npublic void nodeRemoved(DruidNodeDiscovery.Node node) {\n  liveNodes.remove(node.getHostAndPort()); // idempotent removal\n}","preventionTips":["Make NodeListener.nodeRemoved callbacks idempotent and cheap.","Monitor session expiries to distinguish crashes from planned restarts.","Tune ZK session timeout so transient network blips don't drop healthy nodes."],"tags":["zookeeper","curator","service-discovery","node-offline"],"backgroundTag":"resource-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}