apache/cassandra · error · IllegalStateException

Unable to start because the node is draining.

Error message

Unable to start %s because the node is draining.

What it means

checkServiceAllowedToStart(service) refuses to start a service (e.g. thrift/native transport) while the node is draining (nodetool drain). Drain flushes memtables and closes listening connections; starting services mid-drain would reopen the node inconsistently. This is thrown as IllegalStateException so nodetool can display it directly.

Solutions

  1. Restart the Cassandra process fully; a drained node cannot resume services in-place
  2. Remove drain steps from automation that expect the node to keep serving
  3. Run `nodetool statusbinary`/logs to confirm the node was drained, then perform a full restart
  4. Reconfigure so transports are only started at boot, not via JMX after drain

Example fix

// before: restart service via JMX after drain (fails)
ssproxy.startRPCServer();
// after: full process restart instead of starting services post-drain
sudo systemctl restart cassandra
Defensive patterns

Strategy: try-catch

Validate before calling

// check via JMX before starting services
if (storageService.isDraining()) throw new SkipRestartException("node draining");

Try / catch

try { storageService.startRPCServer(); } catch (IllegalStateException e) { if (e.getMessage().contains("draining")) { scheduleFullRestart(); } else { throw e; } }

Prevention

When it happens

Trigger: Calling startRPCServer()/native transport start (or nodetool enablebinary etc., or service restart) after or during `nodetool drain`.

Common situations: Automated restart scripts or service managers (systemd) restarting transport while a drain is in progress or after a drain without restarting the whole JVM.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors


AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/531ed5ce6247e4e7. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/service/StorageService.java:4157

    /**
     * Remove a postshutdownhook
     */
    public synchronized boolean removePostShutdownHook(Runnable hook)
    {
        return postShutdownHooks.remove(hook);
    }

    /**
     * Some services are shutdown during draining and we should not attempt to start them again.
     *
     * @param service - the name of the service we are trying to start.
     * @throws IllegalStateException - an exception that nodetool is able to convert into a message to display to the user
     */
    synchronized void checkServiceAllowedToStart(String service)
    {
        if (isDraining()) // when draining isShutdown is also true, so we check first to return a more accurate message
            throw new IllegalStateException(String.format("Unable to start %s because the node is draining.", service));

        if (isShutdown()) // do not rely on operationMode in case it gets changed to decommissioned or other
            throw new IllegalStateException(String.format("Unable to start %s because the node was drained.", service));

        if (!isNormal() && joinRing) // if the node is not joining the ring, it is gossipping-only member which is in STARTING state forever
            throw new IllegalStateException(String.format("Unable to start %s because the node is not in the normal state.", service));
    }

    // Never ever do this at home. Used by tests.
    @VisibleForTesting
    public IPartitioner setPartitionerUnsafe(IPartitioner newPartitioner)
    {
        checkNotNull(newPartitioner, "newPartitioner is null");
        checkState(originalPartitioner == null, "Already changed the partitioner without resetting");
        originalPartitioner = DatabaseDescriptor.setPartitionerUnsafe(newPartitioner);
        valueFactory = new VersionedValue.VersionedValueFactory(newPartitioner);
        return originalPartitioner;
    }

View on GitHub (pinned to 88fd0f6a0e)