{"record":{"id":"72a08053fadd7751","repo":"apache/cassandra","slug":"endpoints-by-dc-differ","errorCode":null,"errorMessage":"Endpoints by dc differ: {} != {}","messagePattern":"Endpoints by dc differ: (.+?) != (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/tcm/membership/Directory.java","lineNumber":921,"sourceCode":"            logger.warn(\"Last modified differ: {} != {}\", lastModified, other.lastModified);\n        }\n        if (!Objects.equals(peers, other.peers))\n        {\n            logger.warn(\"Peers differ: {} != {}\", peers, other.peers);\n            dumpDiff(logger, peers, other.peers);\n        }\n        if (!Objects.equals(locations, other.locations))\n        {\n            logger.warn(\"Locations differ: {} != {}\", locations, other.locations);\n        }\n        if (!Objects.equals(states, other.states))\n        {\n            logger.warn(\"States differ: {} != {}\", states, other.states);\n            dumpDiff(logger, states, other.states);\n        }\n        if (!Objects.equals(endpointsByDC, other.endpointsByDC))\n        {\n            logger.warn(\"Endpoints by dc differ: {} != {}\", endpointsByDC, other.endpointsByDC);\n            dumpDiff(logger, endpointsByDC.asMap(), other.endpointsByDC.asMap());\n        }\n        if (!Objects.equals(racksByDC, other.racksByDC))\n        {\n            logger.warn(\"Racks by dc differ: {} != {}\", racksByDC, other.racksByDC);\n        }\n        if (!Objects.equals(versions, other.versions))\n        {\n            logger.warn(\"Versions differ: {} != {}\", versions, other.versions);\n            dumpDiff(logger, versions, other.versions);\n        }\n        if (!Objects.equals(addresses, other.addresses))\n        {\n            logger.warn(\"Addresses differ: {} != {}\", addresses, other.addresses);\n            dumpDiff(logger, addresses, other.addresses);\n        }\n    }\n","sourceCodeStart":903,"sourceCodeEnd":939,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/membership/Directory.java#L903-L939","documentation":"WARN log from Directory.dumpDiff(Directory) indicating the endpointsByDC index (dc to endpoints mapping derived from locations) differs between the two Directory snapshots. This index is derived state, so divergence usually follows from a peers or locations difference; it changes DC-aware routing and replication calculations. Diagnostic only.","triggerScenarios":"dumpDiff(other) when the endpointsByDC Multimap is unequal; typically seen alongside a Locations differ warning, e.g. after a node was added to or moved between datacenters in only one view.","commonSituations":"Topology change (node added to a DC) visible to one member but not the other; stale snapshot comparison after a DC expansion; split-brain with independent topology changes.","solutions":["Fix the underlying peers/locations divergence first; endpointsByDC is derived and will follow","Replay missed metadata log entries on the lagging node","Verify both nodes agree on the cluster topology by comparing directory snapshots taken at the same epoch","Rebuild the diverging node's cluster metadata if simple replay does not converge"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// endpointsByDC is derived; validate the source maps instead\nif (java.util.Objects.equals(dirA.locations(), dirB.locations()))\n    // then endpointsByDC should match; only now is a diff meaningful\n    dirA.dumpDiff(dirB);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Fix peers/locations divergence first; derived indexes converge automatically","After DC expansion, wait for all nodes to commit/observe the topology change","Compare snapshots at the same epoch","Keep a single metadata quorum to avoid divergent derived views"],"tags":["cassandra","tcm","topology","diagnostics"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}