apache/cassandra · error · IllegalStateException

Node is still rebuilding. Check nodetool netstats.

Error message

Node is still rebuilding. Check nodetool netstats.

What it means

Rebuild() uses an AtomicBoolean `isRebuilding` compareAndSet as a latch: only one rebuild may run at a time per node. If a rebuild is already in progress (started via nodetool or JMX), a second call throws this IllegalStateException immediately.

Source

Thrown at src/java/org/apache/cassandra/service/Rebuild.java:80

public class Rebuild
{
    private static final AtomicBoolean isRebuilding = new AtomicBoolean();

    private static final Logger logger = LoggerFactory.getLogger(Rebuild.class);

    @VisibleForTesting
    public static void unsafeResetRebuilding()
    {
        isRebuilding.set(false);
    }

    public static void rebuild(String sourceDc, String keyspace, String tokens, String specificSources, boolean excludeLocalDatacenterNodes)
    {
        // check ongoing rebuild
        if (!isRebuilding.compareAndSet(false, true))
        {
            throw new IllegalStateException("Node is still rebuilding. Check nodetool netstats.");
        }

        if (sourceDc != null)
        {
            if (sourceDc.equals(DatabaseDescriptor.getLocalDataCenter()) && excludeLocalDatacenterNodes) // fail if source DC is local and --exclude-local-dc is set
                throw new IllegalArgumentException("Cannot set source data center to be local data center, when excludeLocalDataCenter flag is set");
            Set<String> availableDCs = ClusterMetadata.current().directory.knownDatacenters();
            if (!availableDCs.contains(sourceDc))
            {
                throw new IllegalArgumentException(String.format("Provided datacenter '%s' is not a valid datacenter, available datacenters are: %s",
                                                                 sourceDc, String.join(",", availableDCs)));
            }
        }

        try
        {
            // check the arguments
            if (keyspace == null && tokens != null)

View on GitHub (pinned to 88fd0f6a0e)

Solutions

  1. Check progress with `nodetool netstats` and wait for the current rebuild's streaming to finish before retrying
  2. Inspect logs for the active rebuild / StreamSession activity; if it is genuinely stuck, restart the node (which clears the in-memory flag) and rebuild again
  3. Serialize rebuild calls in automation (e.g. lock or poll until netstats shows no streams)

Example fix

// before: immediate retry on failure
nodetool rebuild -- source_dc
// after: wait until no active streams
while [ "$(nodetool netstats | grep -c 'Receiving' || true)" -gt 0 ]; do sleep 60; done
nodetool rebuild -- source_dc
Defensive patterns

Strategy: validation

Validate before calling

// Java/JMX caller
if (StorageService.instance.isRebuilding())
    throw new IllegalStateException("rebuild already in progress; wait for completion");
StorageService.instance.rebuild(sourceDc, keyspace, tokens, specificSources, excludeLocalDc);

Try / catch

try { rebuild(...); } catch (IllegalStateException e) { if (e.getMessage().contains("still rebuilding")) scheduleRetryAfterNetstatsIdle(); else throw e; }

Prevention

When it happens

Trigger: Calling StorageService.rebuild(...) (or `nodetool rebuild`) while a previous rebuild on the same node has not yet completed.

Common situations: Operator retries `nodetool rebuild` because the first run appears hung; automation scripts fire rebuild concurrently; a long streaming rebuild from a large source DC is still running.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/93c25175ed0d33ab. Report an issue: GitHub.