apache/cassandra · error · IllegalStateException

Couldn't commit the transformation. Is the node shutting dow

Error message

Couldn't commit the transformation. Is the node shutting down?

What it means

ClusterMetadataService.commit() blocks waiting for the committed transformation's epoch to be enacted locally. The wait can be interrupted, and this library converts that InterruptedException into an IllegalStateException with this message, preserving the cause. It signals that the commit did complete (or fail) but the caller thread's wait was interrupted — most commonly during node shutdown.

Source

Thrown at src/java/org/apache/cassandra/tcm/ClusterMetadataService.java:717

                TCMMetrics.instance.commitSuccessLatency.update(nanoTime() - startTime, NANOSECONDS);
                return onSuccess.accept(awaitAtLeast(result.success().epoch));
            }
            else
            {
                TCMMetrics.instance.recordCommitFailureLatency(nanoTime() - startTime, NANOSECONDS, result.failure().rejected);
                logger.debug("Failed to commit {} after {} attempts ({}): {} {}",
                             transform.kind(), retryPolicy.attempts(), retryPolicy,
                             result.failure().code, result.failure().message);
                return onFailure.accept(result.failure().code, result.failure().message);
            }
        }
        catch (TimeoutException t)
        {
            throw new IllegalStateException(String.format("Timed out while waiting for the follower to enact the epoch %s", result.success().epoch), t);
        }
        catch (InterruptedException e)
        {
            throw new IllegalStateException("Couldn't commit the transformation. Is the node shutting down?", e);
        }
    }

    private static Retry getRetryPolicy(Transformation.Kind kind)
    {
        Retry retryPolicy;
        if (kind == Transformation.Kind.STARTUP)
        {
            retryPolicy = Retry.withNoTimeLimit(TCMMetrics.instance.commitRetries, Retry.unsafeRetryIndefinitely());
        }
        else if (kind == Transformation.Kind.SCHEMA_CHANGE)
        {
            long deadlineNanos = nanoTime() + DatabaseDescriptor.getRpcTimeout(TimeUnit.NANOSECONDS);
            retryPolicy = Retry.until(deadlineNanos, TCMMetrics.instance.commitRetries);
        }
        else
        {
            // On non-CMS members, which send commit requests via messaging to the CMS members, the exponential backoff

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Retry the operation after ensuring the node is not shutting down; check StorageService operation mode before committing
Do not swallow InterruptedException — if you wrap the commit call, re-interrupt the thread (Thread.currentThread().interrupt()) before retrying
Re-run the transformation after restart; TCM log commits are retried safely since transformation state is persisted in the cluster metadata log
If seen during startup/shutdown races in tests, use test framework hooks to await node readiness before committing

Example fix

// before
ClusterMetadataService.instance.commit(transform, ok -> ok, (c, m) -> { throw new IllegalStateException(m); });
// after
if (!StorageService.instance.isStarting() && !StorageService.instance.isShutdown())
{
    try
    {
        ClusterMetadataService.instance.commit(transform, ok -> ok, (c, m) -> { throw new IllegalStateException(m); });
    }
    catch (IllegalStateException e)
    {
        if (e.getCause() instanceof InterruptedException)
            Thread.currentThread().interrupt(); // re-interrupt before retry/abort
        throw e;
    }
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (StorageService.instance.isShutdown() || StorageService.instance.isStarting())
    throw new IllegalStateException("Node is shutting down/startup; skip commit");

Type guard

boolean canCommit = !Thread.currentThread().isInterrupted()
                   && !StorageService.instance.isShutdown();

Try / catch

try
{
    ClusterMetadataService.instance.commit(transform, ok -> ok, (c, m) -> { throw new IllegalStateException(m); });
}
catch (IllegalStateException e)
{
    if (e.getCause() instanceof InterruptedException)
    {
        Thread.currentThread().interrupt();
        // abort or retry after confirming node is up
    }
    else throw e;
}

Prevention

Check node operation mode before issuing commits during drain/stop workflows
Never swallow InterruptedException in wrapper code around TCM commits
Order shutdown so metadata-mutating work completes before interrupting worker threads
In tests, await node readiness/initialisation before triggering transformations

When it happens

Trigger: Calling ClusterMetadataService.commit(transform, onSuccess, onFailure) from a thread that gets interrupted while waiting in awaitAtLeast(result.success().epoch) after a successful commit; typical when the node is draining/shutting down and the calling thread is interrupted.

Common situations: Node shutdown or drain while schema changes or topology changes (e.g. REMOVENODE, bootstrap) are being committed; test teardown interrupting threads; executors cancelled mid-commit.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/b91eaac2ab50d37d. Report an issue: GitHub.