apache/cassandra · warning

Last modified differ: {} != {}

Error message

Last modified differ: {} != {}

What it means

A WARN log from Directory.dumpDiff(Directory) indicating the two compared Directory snapshots have different lastModified timestamps. lastModified records when the directory was last changed, so differing values signal the snapshots were taken at different points in the cluster metadata history. It is informational divergence output, not an exception.

Source

Thrown at src/java/org/apache/cassandra/tcm/membership/Directory.java:903

               Objects.equals(states, directory.states) &&
               Objects.equals(endpointsByDC, directory.endpointsByDC) &&
               Objects.equals(racksByDC, directory.racksByDC) &&
               Objects.equals(versions, directory.versions) &&
               Objects.equals(addresses, directory.addresses) &&
               Objects.equals(removedNodes, directory.removedNodes);
    }
    
    private static final Logger logger = LoggerFactory.getLogger(Directory.class);

    public void dumpDiff(Directory other)
    {
        if (nextId != other.nextId)
        {
            logger.warn("nextId differ: {} != {}", nextId, other.nextId);
        }
        if (!Objects.equals(lastModified, other.lastModified))
        {
            logger.warn("Last modified differ: {} != {}", lastModified, other.lastModified);
        }
        if (!Objects.equals(peers, other.peers))
        {
            logger.warn("Peers differ: {} != {}", peers, other.peers);
            dumpDiff(logger, peers, other.peers);
        }
        if (!Objects.equals(locations, other.locations))
        {
            logger.warn("Locations differ: {} != {}", locations, other.locations);
        }
        if (!Objects.equals(states, other.states))
        {
            logger.warn("States differ: {} != {}", states, other.states);
            dumpDiff(logger, states, other.states);
        }
        if (!Objects.equals(endpointsByDC, other.endpointsByDC))
        {
            logger.warn("Endpoints by dc differ: {} != {}", endpointsByDC, other.endpointsByDC);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Ensure both nodes have caught up to the same cluster metadata epoch before comparing directories
2. Force metadata catch-up by restarting the lagging node so it replays the full cluster metadata log from the CMS
3. Check cluster connectivity so the node can fetch the latest cluster metadata commits
4. If comparing snapshots, take both snapshots at a synchronized epoch

Example fix

// before: diffing unsynchronized snapshots
nodeA.directory.dumpDiff(nodeB.directory);
// after: wait for both to reach the same epoch
ClusterMetadata.sync(nodeA, nodeB.metadata().epoch);
nodeA.directory.dumpDiff(nodeB.directory);

// Synchronize both nodes to the same epoch before comparing directories
Epoch target = nodeB.clusterMetadata().epoch;
ClusterMetadataService.instance().fetchLogAndWait(target, java.time.Duration.ofSeconds(30));
if (!nodeA.directory.lastModified().equals(nodeB.directory.lastModified()))
    throw new IllegalStateException("Snapshots not at the same epoch");

boolean samePointInTime(Directory a, Directory b) {
    return java.util.Objects.equals(a.lastModified(), b.lastModified());
}

When it happens

Trigger: Calling dumpDiff(other) where this.lastModified is not Objects.equals to other.lastModified; usually accompanies other divergences (peers, states) when a node's cluster metadata lags or diverges from another's.

Common situations: A node restarting and loading an older persisted Directory while the rest of the cluster advanced; snapshots pulled from different nodes at different times during debugging; TCM log replay gaps after network partitions.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• nextId differ: {} != {}
• Peers differ: {} != {}
• Locations differ: {} != {}
• States differ: {} != {}
• Endpoints by dc differ: {} != {}

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/16fe74d6d1fd95a4. Report an issue: GitHub.