apache/cassandra · warning

Endpoints by dc differ: {} != {}

Error message

Endpoints by dc differ: {} != {}

What it means

WARN log from Directory.dumpDiff(Directory) indicating the endpointsByDC index (dc to endpoints mapping derived from locations) differs between the two Directory snapshots. This index is derived state, so divergence usually follows from a peers or locations difference; it changes DC-aware routing and replication calculations. Diagnostic only.

Source

Thrown at src/java/org/apache/cassandra/tcm/membership/Directory.java:921

            logger.warn("Last modified differ: {} != {}", lastModified, other.lastModified);
        }
        if (!Objects.equals(peers, other.peers))
        {
            logger.warn("Peers differ: {} != {}", peers, other.peers);
            dumpDiff(logger, peers, other.peers);
        }
        if (!Objects.equals(locations, other.locations))
        {
            logger.warn("Locations differ: {} != {}", locations, other.locations);
        }
        if (!Objects.equals(states, other.states))
        {
            logger.warn("States differ: {} != {}", states, other.states);
            dumpDiff(logger, states, other.states);
        }
        if (!Objects.equals(endpointsByDC, other.endpointsByDC))
        {
            logger.warn("Endpoints by dc differ: {} != {}", endpointsByDC, other.endpointsByDC);
            dumpDiff(logger, endpointsByDC.asMap(), other.endpointsByDC.asMap());
        }
        if (!Objects.equals(racksByDC, other.racksByDC))
        {
            logger.warn("Racks by dc differ: {} != {}", racksByDC, other.racksByDC);
        }
        if (!Objects.equals(versions, other.versions))
        {
            logger.warn("Versions differ: {} != {}", versions, other.versions);
            dumpDiff(logger, versions, other.versions);
        }
        if (!Objects.equals(addresses, other.addresses))
        {
            logger.warn("Addresses differ: {} != {}", addresses, other.addresses);
            dumpDiff(logger, addresses, other.addresses);
        }
    }

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Fix the underlying peers/locations divergence first; endpointsByDC is derived and will follow
2. Replay missed metadata log entries on the lagging node
3. Verify both nodes agree on the cluster topology by comparing directory snapshots taken at the same epoch
4. Rebuild the diverging node's cluster metadata if simple replay does not converge

// endpointsByDC is derived; validate the source maps instead
if (java.util.Objects.equals(dirA.locations(), dirB.locations()))
    // then endpointsByDC should match; only now is a diff meaningful
    dirA.dumpDiff(dirB);

When it happens

Trigger: dumpDiff(other) when the endpointsByDC Multimap is unequal; typically seen alongside a Locations differ warning, e.g. after a node was added to or moved between datacenters in only one view.

Common situations: Topology change (node added to a DC) visible to one member but not the other; stale snapshot comparison after a DC expansion; split-brain with independent topology changes.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Locations differ: {} != {}
• Racks by dc differ: {} != {}
• nextId differ: {} != {}
• Last modified differ: {} != {}
• Peers differ: {} != {}

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/72a08053fadd7751. Report an issue: GitHub.