{"record":{"id":"ac4d8be306424174","repo":"apache/cassandra","slug":"can-not-fetch-log-entries-during-shutdown","errorCode":null,"errorMessage":"Can not fetch log entries during shutdown","messagePattern":"Can not fetch log entries during shutdown","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/tcm/PeerLogFetcher.java","lineNumber":67,"sourceCode":"        this.log = log;\n    }\n\n    /**\n     * fetch log entries from the given remote, we have already seen a message from this replica with epoch awaitAtleast.\n     */\n    public ClusterMetadata fetchLogEntriesAndWait(InetAddressAndPort remote, Epoch awaitAtleast)\n    {\n        ClusterMetadata metadata = ClusterMetadata.current();\n        if (metadata.epoch.isEqualOrAfter(awaitAtleast))\n            return metadata;\n\n        try\n        {\n            return asyncFetchLog(remote, awaitAtleast).get(DatabaseDescriptor.getRpcTimeout(TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS);\n        }\n        catch (InterruptedException e)\n        {\n            throw new RuntimeException(\"Can not fetch log entries during shutdown\", e);\n        }\n        catch (ExecutionException | TimeoutException e)\n        {\n            logger.warn(\"Could not fetch log entries from peer, remote = {}, await = {}\", remote, awaitAtleast);\n            logger.debug(\"Exception while fetching log entries from peer, remote = {}\", remote, e);\n        }\n        return metadata;\n    }\n\n    public Future<ClusterMetadata> asyncFetchLog(InetAddressAndPort remote, Epoch awaitAtleast)\n    {\n        return EpochAwareDebounce.instance.getAsync(() -> fetchLogEntriesAndWaitInternal(remote, awaitAtleast), awaitAtleast);\n    }\n\n    private Future<ClusterMetadata> fetchLogEntriesAndWaitInternal(InetAddressAndPort remote, Epoch awaitAtleast)\n    {\n        Epoch before = ClusterMetadata.current().epoch;\n        if (before.isEqualOrAfter(awaitAtleast))","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/PeerLogFetcher.java#L49-L85","documentation":"PeerLogFetcher.fetchLogEntriesAndWait performs a blocking get() on the async log fetch future with the RPC timeout. If the future completes exceptionally because the fetch task was interrupted (typically during node shutdown when the executor is terminating), it is rethrown as RuntimeException(\"Can not fetch log entries during shutdown\", e). It signals that log catch-up from a peer cannot proceed because this node is shutting down.","triggerScenarios":"Calling fetchLogEntriesAndWait while the node's executor/stage is shutting down, so asyncFetchLog(remote, awaitAtleast)'s future completes with InterruptedException; the get() throws ExecutionException wrapping it and the catch at PeerLogFetcher.java:67 converts it to RuntimeException.","commonSituations":"Graceful decommission/stop racing with a background TCM log catch-up; tests stopping a cluster node mid-fetch; shutdown hook triggering metadata synchronization that then fails because the messaging/execution service is already terminated.","solutions":["Treat as expected during shutdown: gate fetchLogEntriesAndWait on node lifecycle state and skip fetching once shutdown has begun.","Retry the fetch after restart; the node will catch up on the log when it comes back up.","If it occurs outside shutdown, inspect the wrapped InterruptedException cause for the executor that was interrupted.","In tests, ensure the node is fully started and not being torn down before triggering log catch-up."],"exampleFix":"// before\nreturn asyncFetchLog(remote, awaitAtleast).get(timeout, MILLISECONDS);\n// after\nif (StorageService.instance.isShutdownStarted())\n{\n    logger.debug(\"Skipping log fetch from {}: shutdown in progress\", remote);\n    return null;\n}\nreturn asyncFetchLog(remote, awaitAtleast).get(timeout, MILLISECONDS);","handlingStrategy":"try-catch","validationCode":"if (StorageService.instance.isShutdownStarted() || StorageService.instance.isStarting())\n    return; // don't fetch during shutdown/startup transitions","typeGuard":null,"tryCatchPattern":"try { entries = fetcher.fetchLogEntriesAndWait(remote, epoch); } catch (RuntimeException e) { if (e.getMessage().contains(\"during shutdown\")) logger.debug(\"Skipping log fetch: node shutting down\"); else throw e; }","preventionTips":["Check node lifecycle state before initiating log catch-up","Catch and downgrade this exception in shutdown paths and background catch-up tasks","In tests, stop background fetchers before tearing down nodes"],"tags":["tcm","shutdown","interrupted","log-fetch"],"backgroundTag":"request-timeout","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}