apache/cassandra · warning

Racks by dc differ: {} != {}

Error message

Racks by dc differ: {} != {}

What it means

WARN log from Directory.dumpDiff(Directory) indicating the racksByDC mapping (dc to set of racks) differs between two Directory snapshots. Like endpointsByDC this is derived placement state; divergence means the views disagree on which racks exist per datacenter, affecting rack-aware replica placement. Diagnostic output only.

Source

Thrown at src/java/org/apache/cassandra/tcm/membership/Directory.java:926

            dumpDiff(logger, peers, other.peers);
        }
        if (!Objects.equals(locations, other.locations))
        {
            logger.warn("Locations differ: {} != {}", locations, other.locations);
        }
        if (!Objects.equals(states, other.states))
        {
            logger.warn("States differ: {} != {}", states, other.states);
            dumpDiff(logger, states, other.states);
        }
        if (!Objects.equals(endpointsByDC, other.endpointsByDC))
        {
            logger.warn("Endpoints by dc differ: {} != {}", endpointsByDC, other.endpointsByDC);
            dumpDiff(logger, endpointsByDC.asMap(), other.endpointsByDC.asMap());
        }
        if (!Objects.equals(racksByDC, other.racksByDC))
        {
            logger.warn("Racks by dc differ: {} != {}", racksByDC, other.racksByDC);
        }
        if (!Objects.equals(versions, other.versions))
        {
            logger.warn("Versions differ: {} != {}", versions, other.versions);
            dumpDiff(logger, versions, other.versions);
        }
        if (!Objects.equals(addresses, other.addresses))
        {
            logger.warn("Addresses differ: {} != {}", addresses, other.addresses);
            dumpDiff(logger, addresses, other.addresses);
        }
    }

    public static <K, V> void dumpDiff(Logger logger, Map<K, V> l, Map<K, V> r)
    {
        for (K k : Sets.intersection(l.keySet(), r.keySet()))
        {
            V lv = l.get(k);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Resolve the underlying peers/locations divergence; racksByDC is derived and will converge with it
2. Ensure every node's snitch reports the same dc/rack names for the same endpoints (check cassandra-rackdc.properties)
3. Replay missed cluster metadata commits on the diverging node
4. Re-fetch the current Directory from the CMS before re-running the comparison

// Ensure snitch-derived dc/rack names are consistent before comparing
for (InetAddressAndPort ep : allEndpoints)
    assert snitch.getDatacenter(ep).equals(expectedDc(ep))
        : "Snitch mismatch for " + ep;

When it happens

Trigger: dumpDiff(other) when racksByDC maps are unequal; typically a consequence of a node registering in a new rack/dc in one view but not the other, or a node disappearing from a rack in one view.

Common situations: Rack added or removed as part of a topology change seen by only part of the cluster; stale directory snapshot compared against a current one; misconfigured snitch reporting inconsistent rack names across nodes.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Locations differ: {} != {}
• Endpoints by dc differ: {} != {}
• nextId differ: {} != {}
• Last modified differ: {} != {}
• Peers differ: {} != {}

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/6a8c259ec2e3ef21. Report an issue: GitHub.