{"record":{"id":"803c10fadfc0c7b8","repo":"apache/cassandra","slug":"still-behind-after-fetching-log-from-cms","errorCode":null,"errorMessage":"Still behind after fetching log from CMS","messagePattern":"Still behind after fetching log from CMS","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/ClusterMetadataService.java","lineNumber":982,"sourceCode":"            return metadata;\n\n        if (log.isPaused())\n        {\n            logger.debug(\"Fetch metadata log from peer or CMS was requested, but log processing is paused\");\n            return metadata;\n        }\n\n        Epoch before = metadata.epoch;\n        if (before.isEqualOrAfter(awaitAtLeast))\n            return metadata;\n\n        metadata = fetchLogFromPeer(metadata, from, awaitAtLeast);\n        if (metadata.epoch.isEqualOrAfter(awaitAtLeast))\n            return metadata;\n\n        metadata = fetchLogFromCMS(awaitAtLeast);\n        if (metadata.epoch.isBefore(awaitAtLeast))\n            throw new IllegalStateException(\"Still behind after fetching log from CMS\");\n        logger.debug(\"Fetched log from CMS - caught up from epoch {} to epoch {}\", before, metadata.epoch);\n        return metadata;\n    }\n\n    /**\n     * Combines {@link #fetchLogFromPeer} with {@link #fetchLogFromCMS} to synchronously fetch and apply log entries\n     * up to the requested epoch. The supplied peer will be contacted first and if after doing so, the current local\n     * metadata is not caught up to at least the required epoch, a further request is made to the CMS.\n     * The returned ClusterMetadata is guaranteed to have been published, though it may have also been superceded by\n     * further updates.\n     * If the requested epoch is not reached even after fetching from the CMS, an IllegalStateException is thrown.\n     * @param from Initial peer to contact. Usually this is the sender of a message containing the requested epoch,\n     *             which means it can be assumed that this peer (if available) can supply any missing log entries.\n     * @param awaitAtLeast The requested epoch.\n     * @return A published ClusterMetadata with all entries up to (at least) the requested epoch enacted.\n     * @throws IllegalStateException if the requested epoch could not be reached, even after falling back to CMS catchup\n     */\n    public ClusterMetadata fetchLogFromPeerOrCMS(InetAddressAndPort from, Epoch awaitAtLeast)","sourceCodeStart":964,"sourceCodeEnd":1000,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/ClusterMetadataService.java#L964-L1000","documentation":"catchup() first tries fetching the log from a peer replica, then from the CMS itself. If after both fetches the local applied epoch is still before the requested epoch (awaitAtLeast), it gives up and throws this IllegalStateException. It indicates the node could not converge on cluster metadata through either path.","triggerScenarios":"Calling a path that triggers catchup(metadata, from, awaitAtLeast) where fetchLogFromPeer returns an epoch before awaitAtLeast AND fetchLogFromCMS(awaitAtLeast) also returns metadata.epoch.isBefore(awaitAtLeast).","commonSituations":"Whole-cluster metadata lag (all replicas behind requested epoch); partition where neither peer nor CMS can serve newer entries; persistent replay failure applying fetched entries; misconfigured timeout values during heavy topology operations.","solutions":["Check whether the requested epoch actually exists cluster-wide — if the CMS also lags, the epoch may never have been committed; retry after the cluster stabilises","Inspect logs from the CMS members for commit failures or Paxos conflicts preventing newer epochs","Verify network reachability between the node, peers, and CMS members","Restart the node to force a fresh metadata init and catch-up sequence"],"exampleFix":"// operational check before operations that require catch-up\nClusterMetadata metadata = ClusterMetadata.current();\nif (metadata.epoch.isBefore(EPOCH_REQUIRED))\n{\n    // wait/retry instead of forcing the operation\n    Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS);\n}\n// instead of proceeding directly and hitting 'Still behind after fetching log from CMS'","handlingStrategy":"retry","validationCode":"if (!ClusterMetadataService.instance.isCurrentCMSMember() && !ClusterMetadataService.instance.isInitialized())\n    awaitInitialization(); // don't proceed until fully caught up","typeGuard":"boolean readyForOp = (Epoch required) -> ClusterMetadata.current().epoch.isEqualOrAfter(required);","tryCatchPattern":"try\n{\n    metadata = catchup(metadata, from, awaitAtLeast);\n}\ncatch (IllegalStateException e)\n{\n    if (\"Still behind after fetching log from CMS\".equals(e.getMessage()))\n    {\n        // exponential backoff and re-check cluster-wide epoch progress\n    }\n    else throw e;\n}","preventionTips":["Wait for full initialisation (awaitClusterMetadata/awaitLocal) before topology operations","Avoid triggering catch-up against epochs that may not exist cluster-wide (e.g. after a rollback)","Keep the cluster network partition-free; verify peer reachability","Stagger topology changes so metadata application is not backlogged"],"tags":["tcm","cms","catch-up","replication","epoch"],"backgroundTag":"metadata-catch-up-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}