apache/druid · info

Node [ ] of role [ ] went offline.

Error message

Node [%s] of role [%s] went offline.

What it means

BaseNodeRoleWatcher.childRemoved() detected that an announced child znode for a known Druid node was deleted in ZK, so the node is logged as going offline and removed from the in-memory node map (removeNode), which then notifies registered listeners. This is the standard notification that a peer service instance left the cluster (shutdown or session loss).

Solutions

1. No action if the peer was intentionally stopped — this is the expected offline notification.
2. If unexpected, check the offline node's logs/host for crash, OOM kill, or loss of ZK connectivity.
3. Review druid.curator.sessionTimeout and network stability if nodes flapping on/off causes churn.
4. Ensure dependent services (e.g. SQL planner, task runners) handle the node-gone listener callback gracefully.

// nodeGone handlers must tolerate removal of nodes they never used
@Override
public void nodeRemoved(DruidNodeDiscovery.Node node) {
  liveNodes.remove(node.getHostAndPort()); // idempotent removal
}

When it happens

Trigger: A ZK CHILD_REMOVED event for a node present in the watcher's map: the peer process stopped gracefully (closes its ephemeral announcement), its ZK session expired after a crash, or ZK cleaned up ephemerals on disconnect.

Common situations: Normal service restarts/scaling-down of middle managers, brokers, or historicals; crash-induced ZK session expiry; network partitions causing session loss; load balancer or ops tooling deleting znodes.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Watcher for node role
• Cache for node role [ ] could not be initialized before…
• Cache initialization for node role
• can't start.
• can't stop.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/e3e6412607fd08d9. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/discovery/BaseNodeRoleWatcher.java:209

        LOGGER.error(
            "Node [%s] of role [%s] removal ignored due to mismatched role (expected role [%s]).",
            druidNode.getDruidNode().getUriToUse(),
            druidNode.getNodeRole().getJsonName(),
            nodeRole.getJsonName()
        );
        return;
      }

      if (skipIfUnknown && !nodes.containsKey(druidNode.getDruidNode().getHostAndPortToUse())) {
        LOGGER.debug(
            "Ignoring removal of node [%s] of role [%s] because it is not known to be present in the cache.",
            druidNode.getDruidNode().getUriToUse(),
            nodeRole.getJsonName()
        );
        return;
      }

      LOGGER.warn("Node [%s] of role [%s] went offline.", druidNode.getDruidNode().getUriToUse(), nodeRole.getJsonName());

      removeNode(druidNode);
    }
  }

  @GuardedBy("lock")
  private void removeNode(DiscoveryDruidNode druidNode)
  {
    DiscoveryDruidNode prev = nodes.remove(druidNode.getDruidNode().getHostAndPortToUse());

    if (prev == null) {
      LOGGER.error(
          "Noticed disappearance of unknown druid node [%s] of role [%s].",
          druidNode.getDruidNode().getUriToUse(),
          druidNode.getNodeRole().getJsonName()
      );
      return;
    }

View on GitHub (pinned to 9b90983fd2)