{"record":{"id":"1c417259200623d0","repo":"apache/druid","slug":"node-s-disappeared-but-was-unknown-for-service-l","errorCode":null,"errorMessage":"Node[%s] disappeared but was unknown for service listener [%s].","messagePattern":"Node\\[(.+?)\\] disappeared but was unknown for service listener \\[(.+?)\\]\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"server/src/main/java/org/apache/druid/discovery/DruidNodeDiscoveryProvider.java","lineNumber":216,"sourceCode":"            }\n            catch (Exception ex) {\n              log.error(ex, \"Listener[%s].nodesAdded(%s) threw exception. Ignored.\", listener, nodesAdded);\n            }\n          }\n        }\n      }\n\n      @Override\n      public void nodesRemoved(Collection<DiscoveryDruidNode> nodesDisappeared)\n      {\n        synchronized (lock) {\n          List<DiscoveryDruidNode> nodesRemoved = new ArrayList<>();\n          for (DiscoveryDruidNode node : nodesDisappeared) {\n            DiscoveryDruidNode prev = nodes.remove(node.getDruidNode().getHostAndPortToUse());\n            if (prev != null) {\n              nodesRemoved.add(node);\n            } else {\n              log.warn(\"Node[%s] disappeared but was unknown for service listener [%s].\", node, service);\n            }\n          }\n\n          if (nodesRemoved.isEmpty()) {\n            // Don't bother listeners with an empty update, it doesn't make sense.\n            return;\n          }\n\n          Collection<DiscoveryDruidNode> unmodifiableNodesRemoved = Collections.unmodifiableCollection(nodesRemoved);\n          for (Listener listener : listeners) {\n            try {\n              listener.nodesRemoved(unmodifiableNodesRemoved);\n            }\n            catch (Exception ex) {\n              log.error(ex, \"Listener[%s].nodesRemoved(%s) threw exception. Ignored.\", listener, nodesRemoved);\n            }\n          }\n        }","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/discovery/DruidNodeDiscoveryProvider.java#L198-L234","documentation":"Log warning in DruidNodeDiscoveryProvider.NodeDiscoveryListener when a node removal event arrives for a hostAndPort that is not present in this listener's node map. The remove() returned null, so nothing was removed and listeners are not notified for that node. This indicates the watcher's view is out of sync with the raw announcement stream.","triggerScenarios":"A node disappeared event is processed for a node that was never added to this listener's map — e.g. it was earlier ignored (missing service, duplicate), or the add was never observed because the watcher was created mid-stream.","commonSituations":"Node that announced a service later removed that service key (role downgrade/config change) so the disappearance doesn't match any tracked node; watcher restarts losing prior state; ZK delivering events out of expected order.","solutions":["Verify the disappearing node previously announced the expected service; check its startup/shutdown logs for config changes.","Restart the service hosting the watcher to rebuild the node map from current ZK state.","Check for ZK session expiry or event reordering; stabilize connectivity.","Ensure nodes do not toggle service keys (e.g. druid.service changes) without restarts of the whole cluster's watchers."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Only react to removals for nodes this listener tracks\nif (!knownNodes.containsKey(node.getDruidNode().getHostAndPortToUse())) {\n  return; // stale/unknown removal, ignore\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep node service keys stable across restarts","Rebuild watcher state from current ZK state after session loss","Monitor for event-ordering anomalies in curator watchers"],"tags":["service-discovery","zookeeper","state-sync"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}