apache/cassandra · warning · RuntimeException

Can not fetch log entries during shutdown

Error message

Can not fetch log entries during shutdown

What it means

PeerLogFetcher.fetchLogEntriesAndWait performs a blocking get() on the async log fetch future with the RPC timeout. If the future completes exceptionally because the fetch task was interrupted (typically during node shutdown when the executor is terminating), it is rethrown as RuntimeException("Can not fetch log entries during shutdown", e). It signals that log catch-up from a peer cannot proceed because this node is shutting down.

Source

Thrown at src/java/org/apache/cassandra/tcm/PeerLogFetcher.java:67

        this.log = log;
    }

    /**
     * fetch log entries from the given remote, we have already seen a message from this replica with epoch awaitAtleast.
     */
    public ClusterMetadata fetchLogEntriesAndWait(InetAddressAndPort remote, Epoch awaitAtleast)
    {
        ClusterMetadata metadata = ClusterMetadata.current();
        if (metadata.epoch.isEqualOrAfter(awaitAtleast))
            return metadata;

        try
        {
            return asyncFetchLog(remote, awaitAtleast).get(DatabaseDescriptor.getRpcTimeout(TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS);
        }
        catch (InterruptedException e)
        {
            throw new RuntimeException("Can not fetch log entries during shutdown", e);
        }
        catch (ExecutionException | TimeoutException e)
        {
            logger.warn("Could not fetch log entries from peer, remote = {}, await = {}", remote, awaitAtleast);
            logger.debug("Exception while fetching log entries from peer, remote = {}", remote, e);
        }
        return metadata;
    }

    public Future<ClusterMetadata> asyncFetchLog(InetAddressAndPort remote, Epoch awaitAtleast)
    {
        return EpochAwareDebounce.instance.getAsync(() -> fetchLogEntriesAndWaitInternal(remote, awaitAtleast), awaitAtleast);
    }

    private Future<ClusterMetadata> fetchLogEntriesAndWaitInternal(InetAddressAndPort remote, Epoch awaitAtleast)
    {
        Epoch before = ClusterMetadata.current().epoch;
        if (before.isEqualOrAfter(awaitAtleast))

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. Treat as expected during shutdown: gate fetchLogEntriesAndWait on node lifecycle state and skip fetching once shutdown has begun.
2. Retry the fetch after restart; the node will catch up on the log when it comes back up.
3. If it occurs outside shutdown, inspect the wrapped InterruptedException cause for the executor that was interrupted.
4. In tests, ensure the node is fully started and not being torn down before triggering log catch-up.

Example fix

// before
return asyncFetchLog(remote, awaitAtleast).get(timeout, MILLISECONDS);
// after
if (StorageService.instance.isShutdownStarted())
{
    logger.debug("Skipping log fetch from {}: shutdown in progress", remote);
    return null;
}
return asyncFetchLog(remote, awaitAtleast).get(timeout, MILLISECONDS);

if (StorageService.instance.isShutdownStarted() || StorageService.instance.isStarting())
    return; // don't fetch during shutdown/startup transitions

try { entries = fetcher.fetchLogEntriesAndWait(remote, epoch); } catch (RuntimeException e) { if (e.getMessage().contains("during shutdown")) logger.debug("Skipping log fetch: node shutting down"); else throw e; }

When it happens

Trigger: Calling fetchLogEntriesAndWait while the node's executor/stage is shutting down, so asyncFetchLog(remote, awaitAtleast)'s future completes with InterruptedException; the get() throws ExecutionException wrapping it and the catch at PeerLogFetcher.java:67 converts it to RuntimeException.

Common situations: Graceful decommission/stop racing with a background TCM log catch-up; tests stopping a cluster node mid-fetch; shutdown hook triggering metadata synchronization that then fails because the messaging/execution service is already terminated.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Couldn't commit the transformation. Is the node shutting dow
• Can not replay during shutdown
• ${executor.name} not terminated
• ${executor} has shut down
• CoordinatorBehindException (read command serialized at later

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/ac4d8be306424174. Report an issue: GitHub.