apache/cassandra · error · IllegalStateException

Still behind after fetching log from CMS

Error message

Still behind after fetching log from CMS

What it means

catchup() first tries fetching the log from a peer replica, then from the CMS itself. If after both fetches the local applied epoch is still before the requested epoch (awaitAtLeast), it gives up and throws this IllegalStateException. It indicates the node could not converge on cluster metadata through either path.

Source

Thrown at src/java/org/apache/cassandra/tcm/ClusterMetadataService.java:982

            return metadata;

        if (log.isPaused())
        {
            logger.debug("Fetch metadata log from peer or CMS was requested, but log processing is paused");
            return metadata;
        }

        Epoch before = metadata.epoch;
        if (before.isEqualOrAfter(awaitAtLeast))
            return metadata;

        metadata = fetchLogFromPeer(metadata, from, awaitAtLeast);
        if (metadata.epoch.isEqualOrAfter(awaitAtLeast))
            return metadata;

        metadata = fetchLogFromCMS(awaitAtLeast);
        if (metadata.epoch.isBefore(awaitAtLeast))
            throw new IllegalStateException("Still behind after fetching log from CMS");
        logger.debug("Fetched log from CMS - caught up from epoch {} to epoch {}", before, metadata.epoch);
        return metadata;
    }

    /**
     * Combines {@link #fetchLogFromPeer} with {@link #fetchLogFromCMS} to synchronously fetch and apply log entries
     * up to the requested epoch. The supplied peer will be contacted first and if after doing so, the current local
     * metadata is not caught up to at least the required epoch, a further request is made to the CMS.
     * The returned ClusterMetadata is guaranteed to have been published, though it may have also been superceded by
     * further updates.
     * If the requested epoch is not reached even after fetching from the CMS, an IllegalStateException is thrown.
     * @param from Initial peer to contact. Usually this is the sender of a message containing the requested epoch,
     *             which means it can be assumed that this peer (if available) can supply any missing log entries.
     * @param awaitAtLeast The requested epoch.
     * @return A published ClusterMetadata with all entries up to (at least) the requested epoch enacted.
     * @throws IllegalStateException if the requested epoch could not be reached, even after falling back to CMS catchup
     */
    public ClusterMetadata fetchLogFromPeerOrCMS(InetAddressAndPort from, Epoch awaitAtLeast)

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Check whether the requested epoch actually exists cluster-wide — if the CMS also lags, the epoch may never have been committed; retry after the cluster stabilises
2. Inspect logs from the CMS members for commit failures or Paxos conflicts preventing newer epochs
3. Verify network reachability between the node, peers, and CMS members
4. Restart the node to force a fresh metadata init and catch-up sequence

Example fix

// operational check before operations that require catch-up
ClusterMetadata metadata = ClusterMetadata.current();
if (metadata.epoch.isBefore(EPOCH_REQUIRED))
{
    // wait/retry instead of forcing the operation
    Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS);
}
// instead of proceeding directly and hitting 'Still behind after fetching log from CMS'

if (!ClusterMetadataService.instance.isCurrentCMSMember() && !ClusterMetadataService.instance.isInitialized())
    awaitInitialization(); // don't proceed until fully caught up

boolean readyForOp = (Epoch required) -> ClusterMetadata.current().epoch.isEqualOrAfter(required);

try
{
    metadata = catchup(metadata, from, awaitAtLeast);
}
catch (IllegalStateException e)
{
    if ("Still behind after fetching log from CMS".equals(e.getMessage()))
    {
        // exponential backoff and re-check cluster-wide epoch progress
    }
    else throw e;
}

When it happens

Trigger: Calling a path that triggers catchup(metadata, from, awaitAtLeast) where fetchLogFromPeer returns an epoch before awaitAtLeast AND fetchLogFromCMS(awaitAtLeast) also returns metadata.epoch.isBefore(awaitAtLeast).

Common situations: Whole-cluster metadata lag (all replicas behind requested epoch); partition where neither peer nor CMS can serve newer entries; persistent replay failure applying fetched entries; misconfigured timeout values during heavy topology operations.

Related errors

• Could not catch up to epoch %s even after fetching log from
• Node %s is not a CMS member in epoch %s; members=%s
• %s is already joining the CMS
• %s has already fully joined the CMS
• %s is not currently joining the CMS

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/803c10fadfc0c7b8. Report an issue: GitHub.