apache/cassandra · error · IllegalStateException

Could not catch up to epoch %s even after fetching log from

Error message

Could not catch up to epoch %s even after fetching log from CMS. Highest seen after fetching is %s.

What it means

fetchLogFromCMS() retrieves the full log from the CMS and applies it, expecting the local replica to reach at least the requested epoch (awaitAtLeast). If, even after fetching directly from the CMS, the highest applied epoch is still behind the target, it throws this IllegalStateException — meaning the CMS itself cannot supply a log reaching the requested epoch or fetched metadata cannot be applied locally.

Source

Thrown at src/java/org/apache/cassandra/tcm/ClusterMetadataService.java:843

        if (awaitAtLeast.isBefore(Epoch.FIRST))
            return metadata;

        Epoch ourEpoch = metadata.epoch;
        if (ourEpoch.isEqualOrAfter(awaitAtLeast))
            return metadata;

        if (log.isPaused())
        {
            logger.debug("Fetch metadata log from CMS was requested, but log processing is paused");
            return metadata;
        }

        Retry deadline = Retry.untilElapsed(getCmsAwaitTimeout().to(TimeUnit.NANOSECONDS), TCMMetrics.instance.fetchLogRetries);
        // responses for ALL withhout knowing we have pending
        metadata = processor.fetchLogAndWait(awaitAtLeast, deadline);
        if (metadata.epoch.isBefore(awaitAtLeast))
        {
            throw new IllegalStateException(String.format("Could not catch up to epoch %s even after fetching log from CMS. Highest seen after fetching is %s.",
                                                          awaitAtLeast, ourEpoch));
        }
        return metadata;
    }

    /**
     * Attempts to asynchronously retrieve log entries from a non-CMS peer.
     * Fetches and applies the log state representing the delta between the current local epoch and the one requested.
     * This is used when a message from a peer contains an epoch higher than the current local epoch. As the sender of
     * the message must have seen and enacted the given epoch, they must (under normal circumstances) be able to supply
     * any entries needed to catch up this node.
     * When the returned future completes, the metadata it provides is the current published metadata at the
     * moment of completion. In the expected case, this will have had any fetched transformations up to the requested
     * epoch applied. If the fetch was unsuccessful (e.g. because the peer was unavailable) it will still be whatever
     * the currently published metadata, but which entries have been enacted cannot be guaranteed.
     * @param from peer to request log entries from
     * @param awaitAtLeast the upper epoch required. It's expected that the peer is able to supply log entries up to at
     *                     least this epoch.

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Verify connectivity to CMS members and that the CMS is healthy (nodetool cms show / describecluster)
Increase cms_await_timeout in cassandra.yaml if the log backlog is large and the fetch legitimately needs longer
Check TCMMetrics.instance.fetchLogRetries and fetch log latency metrics to see whether fetches are retrying/failing
Restart the lagging node so it re-initialises and re-fetches the log from the CMS from a clean state

Example fix

// cassandra.yaml before
cms_await_timeout: 1m
// after (large metadata backlog / slow inter-node links)
cms_await_timeout: 5m

Defensive patterns

Strategy: retry

Validate before calling

ClusterMetadata current = ClusterMetadata.current();
if (current.epoch.isBefore(targetEpoch))
{
    // fetch ahead of the strict call to surface problems early
    ClusterMetadataService.instance.fetchLogFromCMS(targetEpoch);
}

Type guard

boolean caughtUp = (ClusterMetadata cm) -> !cm.epoch.isBefore(targetEpoch);

Try / catch

try
{
    metadata = catchup(path, awaitAtLeast);
}
catch (IllegalStateException e)
{
    if (e.getMessage() != null && e.getMessage().contains("Could not catch up to epoch"))
    {
        // backoff, verify CMS health, then retry
    }
    else throw e;
}

Prevention

Monitor fetchLogRetries and catch-up latency metrics (TCMMetrics) for trends
Set cms_await_timeout generously for clusters with large metadata logs or slow links
Ensure CMS members are highly available and reachable from all nodes
Check CMS health (nodetool cms show) before topology operations

When it happens

Trigger: Awaiting catch-up to an epoch (e.g. during join, CMS placement change, or ClusterMetadataService.initialized) where metadata.epoch.isBefore(awaitAtLeast) even after processor.fetchLogAndWait(awaitAtLeast, deadline) completes within the cms_await_timeout deadline.

Common situations: CMS member unreachable or lagging; network partitions between the node and CMS; cms_await_timeout too low for large metadata backlogs; node stuck replaying a large log during bootstrap.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/1fe0da8476346ef9. Report an issue: GitHub.