{"record":{"id":"4b6313b04e368f34","repo":"apache/cassandra","slug":"queried-for-epoch-s-but-could-not-catch-up-curr","errorCode":null,"errorMessage":"Queried for epoch %s, but could not catch up. Current epoch: %s","messagePattern":"Queried for epoch (.+?), but could not catch up\\. Current epoch: (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/PeerLogFetcher.java","lineNumber":103,"sourceCode":"        if (before.isEqualOrAfter(awaitAtleast))\n        {\n            Promise<ClusterMetadata> res = new AsyncPromise<>();\n            res.setSuccess(ClusterMetadata.current());\n            return res;\n        }\n\n        Promise<LogState> fetchFromRemote = new AsyncPromise<>();\n        Future<ClusterMetadata> appendToLog = fetchFromRemote.map(logState -> {\n            log.append(logState);\n            ClusterMetadata fetched = log.waitForHighestConsecutive();\n            if (fetched.epoch.isEqualOrAfter(awaitAtleast))\n            {\n                TCMMetrics.instance.peerLogEntriesFetched(before, logState.latestEpoch());\n                return fetched;\n            }\n            else\n            {\n                throw new IllegalStateException(String.format(\"Queried for epoch %s, but could not catch up. Current epoch: %s\", awaitAtleast, fetched.epoch));\n            }\n        });\n\n        logger.info(\"Fetching log from {}, at least {}\", remote, awaitAtleast);\n        try (Timer.Context ctx = TCMMetrics.instance.fetchPeerLogLatency.time())\n        {\n            RemoteProcessor.sendWithRetries(Verb.TCM_FETCH_PEER_LOG_REQ,\n                                            new FetchPeerLog(before),\n                                            fetchFromRemote,\n                                            new RemoteProcessor.CandidateIterator(Collections.singletonList(remote), false),\n                                            Retry.untilElapsed(DatabaseDescriptor.getCmsAwaitTimeout().to(TimeUnit.NANOSECONDS), TCMMetrics.instance.fetchLogRetries));\n            return appendToLog;\n        }\n        catch (Throwable t)\n        {\n            fetchFromRemote.cancel(true);\n            appendToLog.cancel(true);\n            JVMStabilityInspector.inspectThrowable(t);","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/PeerLogFetcher.java#L85-L121","documentation":"Thrown by PeerLogFetcher.fetchLogEntriesAndWaitInternal when a replica asks a remote CMS member for log entries starting at a target epoch (awaitAtleast), but the peer's log does not reach that epoch after exhausting retry attempts. The fetcher polls until its local epoch requirement is met; if the remote (or local) log state cannot advance past the current epoch, the retry predicate fails and this IllegalStateException propagates. It signals that the node cannot catch up with cluster metadata from this peer, often because the peer is lagging, unreachable, or the requested epoch does not exist in the peer's log.","triggerScenarios":"Node (re)joining or catching up calls asyncFetchLog -> fetchLogEntriesAndWaitInternal with awaitAtleast set to an epoch the remote peer cannot serve; the remote peer is behind (its latestEpoch < awaitAtleast) and retries time out; a quorum of CMS peers cannot be contacted so fetched epoch never reaches the awaited epoch.","commonSituations":"Joining a cluster whose CMS members are down or partitioned; pointed at stale/old seed nodes after a cluster rebuild; TCM log was truncated/migrated so old epochs no longer exist on reachable peers; severe clock or network partition during bootstrap.","solutions":["Verify the target peer is up and is a current CMS member (nodetool cms describe / show 'list endpoints'); add current CMS endpoints to the contact list.","Check network connectivity/firewall between the node and CMS members on the native/SSL ports.","Confirm the node's configuration (seeds, contact endpoints) points to the live cluster, not an old or rebuilt one.","If the cluster has migrated or truncated the TCM log, ensure nodes bootstrap from an epoch that still exists on peers, or re-bootstrap from seeds.","Restart the joining node once the CMS has advanced and quorum is healthy."],"exampleFix":"// before: joining node cannot reach current CMS peers\nseeds: old-seed-1,old-seed-2   // no longer CMS members\n// after: configure current CMS endpoints\nseeds: cms-node-1.example.com,cms-node-2.example.com","handlingStrategy":"retry","validationCode":"// before joining, check reachable CMS peers can serve the needed epoch\nboolean peerHealthy = ClusterMetadataService.instance().log().latestEpoch().is(FetchedEpoch.required) ;","typeGuard":null,"tryCatchPattern":"catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"could not catch up\")) {\n        // verify CMS membership/health, refresh contact points, then retry with backoff\n        retryWithBackoff(() -> fetchLogEntriesAndWait(remote, awaitAtleast));\n    } else throw e;\n}","preventionTips":["Keep seeds/CMS contact points pointing at live CMS members","Monitor TCMMetrics peerLogEntriesFetched and fetchPeerLogLatency for lag","Ensure CMS quorum health before adding new nodes","Re-check configuration after cluster rebuilds or log migration"],"tags":["tcm","cluster-metadata","bootstrap","retry-exhausted"],"backgroundTag":"request-timeout","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}