apache/cassandra · error · IllegalStateException

Queried for epoch %s, but could not catch up. Current epoch:

Error message

Queried for epoch %s, but could not catch up. Current epoch: %s

What it means

Thrown by PeerLogFetcher.fetchLogEntriesAndWaitInternal when a replica asks a remote CMS member for log entries starting at a target epoch (awaitAtleast), but the peer's log does not reach that epoch after exhausting retry attempts. The fetcher polls until its local epoch requirement is met; if the remote (or local) log state cannot advance past the current epoch, the retry predicate fails and this IllegalStateException propagates. It signals that the node cannot catch up with cluster metadata from this peer, often because the peer is lagging, unreachable, or the requested epoch does not exist in the peer's log.

Source

Thrown at src/java/org/apache/cassandra/tcm/PeerLogFetcher.java:103

        if (before.isEqualOrAfter(awaitAtleast))
        {
            Promise<ClusterMetadata> res = new AsyncPromise<>();
            res.setSuccess(ClusterMetadata.current());
            return res;
        }

        Promise<LogState> fetchFromRemote = new AsyncPromise<>();
        Future<ClusterMetadata> appendToLog = fetchFromRemote.map(logState -> {
            log.append(logState);
            ClusterMetadata fetched = log.waitForHighestConsecutive();
            if (fetched.epoch.isEqualOrAfter(awaitAtleast))
            {
                TCMMetrics.instance.peerLogEntriesFetched(before, logState.latestEpoch());
                return fetched;
            }
            else
            {
                throw new IllegalStateException(String.format("Queried for epoch %s, but could not catch up. Current epoch: %s", awaitAtleast, fetched.epoch));
            }
        });

        logger.info("Fetching log from {}, at least {}", remote, awaitAtleast);
        try (Timer.Context ctx = TCMMetrics.instance.fetchPeerLogLatency.time())
        {
            RemoteProcessor.sendWithRetries(Verb.TCM_FETCH_PEER_LOG_REQ,
                                            new FetchPeerLog(before),
                                            fetchFromRemote,
                                            new RemoteProcessor.CandidateIterator(Collections.singletonList(remote), false),
                                            Retry.untilElapsed(DatabaseDescriptor.getCmsAwaitTimeout().to(TimeUnit.NANOSECONDS), TCMMetrics.instance.fetchLogRetries));
            return appendToLog;
        }
        catch (Throwable t)
        {
            fetchFromRemote.cancel(true);
            appendToLog.cancel(true);
            JVMStabilityInspector.inspectThrowable(t);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Verify the target peer is up and is a current CMS member (nodetool cms describe / show 'list endpoints'); add current CMS endpoints to the contact list.
Check network connectivity/firewall between the node and CMS members on the native/SSL ports.
Confirm the node's configuration (seeds, contact endpoints) points to the live cluster, not an old or rebuilt one.
If the cluster has migrated or truncated the TCM log, ensure nodes bootstrap from an epoch that still exists on peers, or re-bootstrap from seeds.
Restart the joining node once the CMS has advanced and quorum is healthy.

Example fix

// before: joining node cannot reach current CMS peers
seeds: old-seed-1,old-seed-2   // no longer CMS members
// after: configure current CMS endpoints
seeds: cms-node-1.example.com,cms-node-2.example.com

Defensive patterns

Strategy: retry

Validate before calling

// before joining, check reachable CMS peers can serve the needed epoch
boolean peerHealthy = ClusterMetadataService.instance().log().latestEpoch().is(FetchedEpoch.required) ;

Try / catch

catch (IllegalStateException e) {
    if (e.getMessage().contains("could not catch up")) {
        // verify CMS membership/health, refresh contact points, then retry with backoff
        retryWithBackoff(() -> fetchLogEntriesAndWait(remote, awaitAtleast));
    } else throw e;
}

Prevention

Keep seeds/CMS contact points pointing at live CMS members
Monitor TCMMetrics peerLogEntriesFetched and fetchPeerLogLatency for lag
Ensure CMS quorum health before adding new nodes
Re-check configuration after cluster rebuilds or log migration

When it happens

Trigger: Node (re)joining or catching up calls asyncFetchLog -> fetchLogEntriesAndWaitInternal with awaitAtleast set to an epoch the remote peer cannot serve; the remote peer is behind (its latestEpoch < awaitAtleast) and retries time out; a quorum of CMS peers cannot be contacted so fetched epoch never reaches the awaited epoch.

Common situations: Joining a cluster whose CMS members are down or partitioned; pointed at stale/old seed nodes after a cluster rebuild; TCM log was truncated/migrated so old epochs no longer exist on reachable peers; severe clock or network partition during bootstrap.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/4b6313b04e368f34. Report an issue: GitHub.