apache/cassandra · error · IllegalStateException

Can't apply a ClusterMetadata from gossip with epoch

Error message

Can't apply a ClusterMetadata from gossip with epoch 

What it means

applyFromGossip is only valid during the gossip migration window, when the expected ClusterMetadata still has an EMPTY (pre-first) epoch. If the expected metadata's epoch is not before Epoch.EMPTY, the metadata is already committed via TCM and applying a gossip-supplied snapshot would regress state, so it throws IllegalStateException with the offending epoch.

Solutions

1. Stop applying gossip metadata once the node holds a real TCM epoch; check the epoch guard before calling applyFromGossip.
2. Verify the migration ordering: applyFromGossip must run only while expected epoch is EMPTY.
3. If gossip is still propagating stale metadata, complete the upgrade so all peers stop publishing gossip metadata.
4. Restart the migration path cleanly; do not mix gossip application with post-epoch TCM state.

Example fix

// before
service.applyFromGossip(expected, updated); // expected.epoch already > EMPTY
// after
if (expected.epoch.isBefore(Epoch.EMPTY))
    service.applyFromGossip(expected, updated);
else
    logger.warn("Skipping gossip apply; epoch already committed: {}", expected.epoch);

if (!expected.epoch.isBefore(Epoch.EMPTY)) skipGossipApply(expected.epoch);

try { service.applyFromGossip(expected, updated); } catch (IllegalStateException e) { logger.warn("Gossip apply rejected: {}", e.getMessage()); }

When it happens

Trigger: Calling applyFromGossip(expected, updated) where expected.epoch is not isBefore(Epoch.EMPTY) — i.e. the node already has a real TCM-committed epoch, yet a gossip-derived metadata update is attempted (e.g. during or after migration).

Common situations: A node received gossip metadata after it already transitioned to TCM; mixed-mode cluster where some peers still push gossip state; a bug in migration handoff ordering that delivers a stale gossip update post-commit.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Can't apply a ClusterMetadata from gossip when CMSState is…
• All nodes are not yet upgraded
• Another sequence of kind
• Can not add a new in-progress sequence for
• Can not commit transformation

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/a29f248fdf840543. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/ClusterMetadataService.java:539

            return;
        }

        try
        {
            reconfigureCMS(ReplicationParams.meta(metadata));
        }
        catch (Throwable t)
        {
            JVMStabilityInspector.inspectThrowable(t);
            logger.warn("Could not reconfigure CMS, operator should run `nodetool cms reconfigure` to make sure CMS placement is correct", t);
        }
    }

    public boolean applyFromGossip(ClusterMetadata expected, ClusterMetadata updated)
    {
        logger.debug("Applying from gossip, current={} new={}", expected, updated);
        if (!expected.epoch.isBefore(Epoch.EMPTY))
            throw new IllegalStateException("Can't apply a ClusterMetadata from gossip with epoch " + expected.epoch);
        if (state() != GOSSIP)
            throw new IllegalStateException("Can't apply a ClusterMetadata from gossip when CMSState is not GOSSIP: " + state());

        return log.unsafeSetCommittedFromGossip(expected, updated);
    }

    public void setFromGossip(ClusterMetadata fromGossip)
    {
        logger.debug("Setting from gossip, new={}", fromGossip);
        if (state() != GOSSIP)
            throw new IllegalStateException("Can't apply a ClusterMetadata from gossip when CMSState is not GOSSIP: " + state());
        log.unsafeSetCommittedFromGossip(fromGossip);
    }

    public void forceSnapshot(ClusterMetadata snapshot)
    {
        commit(new ForceSnapshot(snapshot));
    }

View on GitHub (pinned to 88fd0f6a0e)