{"record":{"id":"1fe0da8476346ef9","repo":"apache/cassandra","slug":"could-not-catch-up-to-epoch-s-even-after-fetching","errorCode":null,"errorMessage":"Could not catch up to epoch %s even after fetching log from CMS. Highest seen after fetching is %s.","messagePattern":"Could not catch up to epoch (.+?) even after fetching log from CMS\\. Highest seen after fetching is (.+?)\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/ClusterMetadataService.java","lineNumber":843,"sourceCode":"        if (awaitAtLeast.isBefore(Epoch.FIRST))\n            return metadata;\n\n        Epoch ourEpoch = metadata.epoch;\n        if (ourEpoch.isEqualOrAfter(awaitAtLeast))\n            return metadata;\n\n        if (log.isPaused())\n        {\n            logger.debug(\"Fetch metadata log from CMS was requested, but log processing is paused\");\n            return metadata;\n        }\n\n        Retry deadline = Retry.untilElapsed(getCmsAwaitTimeout().to(TimeUnit.NANOSECONDS), TCMMetrics.instance.fetchLogRetries);\n        // responses for ALL withhout knowing we have pending\n        metadata = processor.fetchLogAndWait(awaitAtLeast, deadline);\n        if (metadata.epoch.isBefore(awaitAtLeast))\n        {\n            throw new IllegalStateException(String.format(\"Could not catch up to epoch %s even after fetching log from CMS. Highest seen after fetching is %s.\",\n                                                          awaitAtLeast, ourEpoch));\n        }\n        return metadata;\n    }\n\n    /**\n     * Attempts to asynchronously retrieve log entries from a non-CMS peer.\n     * Fetches and applies the log state representing the delta between the current local epoch and the one requested.\n     * This is used when a message from a peer contains an epoch higher than the current local epoch. As the sender of\n     * the message must have seen and enacted the given epoch, they must (under normal circumstances) be able to supply\n     * any entries needed to catch up this node.\n     * When the returned future completes, the metadata it provides is the current published metadata at the\n     * moment of completion. In the expected case, this will have had any fetched transformations up to the requested\n     * epoch applied. If the fetch was unsuccessful (e.g. because the peer was unavailable) it will still be whatever\n     * the currently published metadata, but which entries have been enacted cannot be guaranteed.\n     * @param from peer to request log entries from\n     * @param awaitAtLeast the upper epoch required. It's expected that the peer is able to supply log entries up to at\n     *                     least this epoch.","sourceCodeStart":825,"sourceCodeEnd":861,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/ClusterMetadataService.java#L825-L861","documentation":"fetchLogFromCMS() retrieves the full log from the CMS and applies it, expecting the local replica to reach at least the requested epoch (awaitAtLeast). If, even after fetching directly from the CMS, the highest applied epoch is still behind the target, it throws this IllegalStateException — meaning the CMS itself cannot supply a log reaching the requested epoch or fetched metadata cannot be applied locally.","triggerScenarios":"Awaiting catch-up to an epoch (e.g. during join, CMS placement change, or ClusterMetadataService.initialized) where metadata.epoch.isBefore(awaitAtLeast) even after processor.fetchLogAndWait(awaitAtLeast, deadline) completes within the cms_await_timeout deadline.","commonSituations":"CMS member unreachable or lagging; network partitions between the node and CMS; cms_await_timeout too low for large metadata backlogs; node stuck replaying a large log during bootstrap.","solutions":["Verify connectivity to CMS members and that the CMS is healthy (nodetool cms show / describecluster)","Increase cms_await_timeout in cassandra.yaml if the log backlog is large and the fetch legitimately needs longer","Check TCMMetrics.instance.fetchLogRetries and fetch log latency metrics to see whether fetches are retrying/failing","Restart the lagging node so it re-initialises and re-fetches the log from the CMS from a clean state"],"exampleFix":"// cassandra.yaml before\ncms_await_timeout: 1m\n// after (large metadata backlog / slow inter-node links)\ncms_await_timeout: 5m","handlingStrategy":"retry","validationCode":"ClusterMetadata current = ClusterMetadata.current();\nif (current.epoch.isBefore(targetEpoch))\n{\n    // fetch ahead of the strict call to surface problems early\n    ClusterMetadataService.instance.fetchLogFromCMS(targetEpoch);\n}","typeGuard":"boolean caughtUp = (ClusterMetadata cm) -> !cm.epoch.isBefore(targetEpoch);","tryCatchPattern":"try\n{\n    metadata = catchup(path, awaitAtLeast);\n}\ncatch (IllegalStateException e)\n{\n    if (e.getMessage() != null && e.getMessage().contains(\"Could not catch up to epoch\"))\n    {\n        // backoff, verify CMS health, then retry\n    }\n    else throw e;\n}","preventionTips":["Monitor fetchLogRetries and catch-up latency metrics (TCMMetrics) for trends","Set cms_await_timeout generously for clusters with large metadata logs or slow links","Ensure CMS members are highly available and reachable from all nodes","Check CMS health (nodetool cms show) before topology operations"],"tags":["tcm","cms","catch-up","epoch","timeout"],"backgroundTag":"metadata-catch-up-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}