apache/druid · warning

Node[ ] disappeared but was unknown for service listener […

Error message

Node[%s] disappeared but was unknown for service listener [%s].

What it means

Log warning in DruidNodeDiscoveryProvider.NodeDiscoveryListener when a node removal event arrives for a hostAndPort that is not present in this listener's node map. The remove() returned null, so nothing was removed and listeners are not notified for that node. This indicates the watcher's view is out of sync with the raw announcement stream.

Solutions

1. Verify the disappearing node previously announced the expected service; check its startup/shutdown logs for config changes.
2. Restart the service hosting the watcher to rebuild the node map from current ZK state.
3. Check for ZK session expiry or event reordering; stabilize connectivity.
4. Ensure nodes do not toggle service keys (e.g. druid.service changes) without restarts of the whole cluster's watchers.

// Only react to removals for nodes this listener tracks
if (!knownNodes.containsKey(node.getDruidNode().getHostAndPortToUse())) {
  return; // stale/unknown removal, ignore
}

When it happens

Trigger: A node disappeared event is processed for a node that was never added to this listener's map — e.g. it was earlier ignored (missing service, duplicate), or the add was never observed because the watcher was created mid-stream.

Common situations: Node that announced a service later removed that service key (role downgrade/config change) so the disappearance doesn't match any tracked node; watcher restarts losing prior state; ZK delivering events out of expected order.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Cache for node role [ ] could not be initialized before…
• Cache initialization for node role
• Node[ ] discovered but already exists [ ].
• Node [ ] of role [ ] went offline.
• Watcher for node role

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/1c417259200623d0. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/discovery/DruidNodeDiscoveryProvider.java:216

            }
            catch (Exception ex) {
              log.error(ex, "Listener[%s].nodesAdded(%s) threw exception. Ignored.", listener, nodesAdded);
            }
          }
        }
      }

      @Override
      public void nodesRemoved(Collection<DiscoveryDruidNode> nodesDisappeared)
      {
        synchronized (lock) {
          List<DiscoveryDruidNode> nodesRemoved = new ArrayList<>();
          for (DiscoveryDruidNode node : nodesDisappeared) {
            DiscoveryDruidNode prev = nodes.remove(node.getDruidNode().getHostAndPortToUse());
            if (prev != null) {
              nodesRemoved.add(node);
            } else {
              log.warn("Node[%s] disappeared but was unknown for service listener [%s].", node, service);
            }
          }

          if (nodesRemoved.isEmpty()) {
            // Don't bother listeners with an empty update, it doesn't make sense.
            return;
          }

          Collection<DiscoveryDruidNode> unmodifiableNodesRemoved = Collections.unmodifiableCollection(nodesRemoved);
          for (Listener listener : listeners) {
            try {
              listener.nodesRemoved(unmodifiableNodesRemoved);
            }
            catch (Exception ex) {
              log.error(ex, "Listener[%s].nodesRemoved(%s) threw exception. Ignored.", listener, nodesRemoved);
            }
          }
        }

View on GitHub (pinned to 9b90983fd2)