{"record":{"id":"73231b3032f3f7ff","repo":"apache/cassandra","slug":"timed-out-while-waiting-for-the-follower-to-enact","errorCode":null,"errorMessage":"Timed out while waiting for the follower to enact the epoch %s","messagePattern":"Timed out while waiting for the follower to enact the epoch (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/ClusterMetadataService.java","lineNumber":713,"sourceCode":"        try\n        {\n            if (result.isSuccess())\n            {\n                TCMMetrics.instance.commitSuccessLatency.update(nanoTime() - startTime, NANOSECONDS);\n                return onSuccess.accept(awaitAtLeast(result.success().epoch));\n            }\n            else\n            {\n                TCMMetrics.instance.recordCommitFailureLatency(nanoTime() - startTime, NANOSECONDS, result.failure().rejected);\n                logger.debug(\"Failed to commit {} after {} attempts ({}): {} {}\",\n                             transform.kind(), retryPolicy.attempts(), retryPolicy,\n                             result.failure().code, result.failure().message);\n                return onFailure.accept(result.failure().code, result.failure().message);\n            }\n        }\n        catch (TimeoutException t)\n        {\n            throw new IllegalStateException(String.format(\"Timed out while waiting for the follower to enact the epoch %s\", result.success().epoch), t);\n        }\n        catch (InterruptedException e)\n        {\n            throw new IllegalStateException(\"Couldn't commit the transformation. Is the node shutting down?\", e);\n        }\n    }\n\n    private static Retry getRetryPolicy(Transformation.Kind kind)\n    {\n        Retry retryPolicy;\n        if (kind == Transformation.Kind.STARTUP)\n        {\n            retryPolicy = Retry.withNoTimeLimit(TCMMetrics.instance.commitRetries, Retry.unsafeRetryIndefinitely());\n        }\n        else if (kind == Transformation.Kind.SCHEMA_CHANGE)\n        {\n            long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(TimeUnit.NANOSECONDS);\n            retryPolicy = Retry.until(deadlineNanos, TCMMetrics.instance.commitRetries);","sourceCodeStart":695,"sourceCodeEnd":731,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/ClusterMetadataService.java#L695-L731","documentation":"After a transformation is committed to the metadata log, the committing node waits for the resulting epoch to be enacted locally/acknowledged (e.g. a follower catching up to the new epoch). If that wait times out, the TimeoutException is converted into an IllegalStateException naming the epoch that failed to be enacted, wrapped as the cause.","triggerScenarios":"commit(transform, onSuccess, onFailure) succeeds in proposing the transformation (result.success() present), but the subsequent wait for the follower/node to enact result.success().epoch exceeds the timeout — e.g. the log processor is stalled, network partitioned, or the node is overloaded.","commonSituations":"CMS leader commits but the local processor stalls under load; network partition between CMS and followers delays catch-up; node shutting down or GC pauses preventing epoch enactment in time; very slow disks replaying the metadata log.","solutions":["Check node/CMS health and network connectivity; retry the operation once the epoch catches up.","Inspect logs around the timeout for stalls in the CommitLogProcessor or metadata log replay.","Increase headroom: reduce load or GC pauses; investigate slow disk/CPU on the committing node.","If the node is shutting down, abort the operation gracefully instead of committing.","Verify cluster quorum is reachable — partitions prevent epoch propagation within the timeout."],"exampleFix":"// before\ntry { service.commit(transform); } // may throw wrapped TimeoutException\n// after\ntry {\n    service.commit(transform);\n} catch (IllegalStateException e) {\n    if (e.getCause() instanceof TimeoutException) {\n        logger.warn(\"Epoch enactment timed out; retrying after catch-up\", e);\n        service.commit(transform); // safe: commit succeeded, waiting on enactment\n    } else throw e;\n}","handlingStrategy":"retry","validationCode":"// Ensure node healthy and connected to CMS before committing\nif (!service.isRunning() || nodeShuttingDown()) skipCommit();","typeGuard":null,"tryCatchPattern":"try { service.commit(transform); }\ncatch (IllegalStateException e) {\n  if (e.getCause() instanceof TimeoutException) retryAfterCatchUp();\n  else throw e;\n}","preventionTips":["Avoid committing during shutdown or heavy GC/load windows","Keep cluster fully connected; resolve partitions quickly","Monitor metadata log replay lag and disk health"],"tags":["timeout","epoch","commit","follower"],"backgroundTag":"request-timeout","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}