{"record":{"id":"93c25175ed0d33ab","repo":"apache/cassandra","slug":"node-is-still-rebuilding-check-nodetool-netstats","errorCode":null,"errorMessage":"Node is still rebuilding. Check nodetool netstats.","messagePattern":"Node is still rebuilding\\. Check nodetool netstats\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/service/Rebuild.java","lineNumber":80,"sourceCode":"\npublic class Rebuild\n{\n    private static final AtomicBoolean isRebuilding = new AtomicBoolean();\n\n    private static final Logger logger = LoggerFactory.getLogger(Rebuild.class);\n\n    @VisibleForTesting\n    public static void unsafeResetRebuilding()\n    {\n        isRebuilding.set(false);\n    }\n\n    public static void rebuild(String sourceDc, String keyspace, String tokens, String specificSources, boolean excludeLocalDatacenterNodes)\n    {\n        // check ongoing rebuild\n        if (!isRebuilding.compareAndSet(false, true))\n        {\n            throw new IllegalStateException(\"Node is still rebuilding. Check nodetool netstats.\");\n        }\n\n        if (sourceDc != null)\n        {\n            if (sourceDc.equals(DatabaseDescriptor.getLocalDataCenter()) && excludeLocalDatacenterNodes) // fail if source DC is local and --exclude-local-dc is set\n                throw new IllegalArgumentException(\"Cannot set source data center to be local data center, when excludeLocalDataCenter flag is set\");\n            Set<String> availableDCs = ClusterMetadata.current().directory.knownDatacenters();\n            if (!availableDCs.contains(sourceDc))\n            {\n                throw new IllegalArgumentException(String.format(\"Provided datacenter '%s' is not a valid datacenter, available datacenters are: %s\",\n                                                                 sourceDc, String.join(\",\", availableDCs)));\n            }\n        }\n\n        try\n        {\n            // check the arguments\n            if (keyspace == null && tokens != null)","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/Rebuild.java#L62-L98","documentation":"Rebuild() uses an AtomicBoolean `isRebuilding` compareAndSet as a latch: only one rebuild may run at a time per node. If a rebuild is already in progress (started via nodetool or JMX), a second call throws this IllegalStateException immediately.","triggerScenarios":"Calling StorageService.rebuild(...) (or `nodetool rebuild`) while a previous rebuild on the same node has not yet completed.","commonSituations":"Operator retries `nodetool rebuild` because the first run appears hung; automation scripts fire rebuild concurrently; a long streaming rebuild from a large source DC is still running.","solutions":["Check progress with `nodetool netstats` and wait for the current rebuild's streaming to finish before retrying","Inspect logs for the active rebuild / StreamSession activity; if it is genuinely stuck, restart the node (which clears the in-memory flag) and rebuild again","Serialize rebuild calls in automation (e.g. lock or poll until netstats shows no streams)"],"exampleFix":"// before: immediate retry on failure\nnodetool rebuild -- source_dc\n// after: wait until no active streams\nwhile [ \"$(nodetool netstats | grep -c 'Receiving' || true)\" -gt 0 ]; do sleep 60; done\nnodetool rebuild -- source_dc","handlingStrategy":"validation","validationCode":"// Java/JMX caller\nif (StorageService.instance.isRebuilding())\n    throw new IllegalStateException(\"rebuild already in progress; wait for completion\");\nStorageService.instance.rebuild(sourceDc, keyspace, tokens, specificSources, excludeLocalDc);","typeGuard":null,"tryCatchPattern":"try { rebuild(...); } catch (IllegalStateException e) { if (e.getMessage().contains(\"still rebuilding\")) scheduleRetryAfterNetstatsIdle(); else throw e; }","preventionTips":["Poll `nodetool netstats` until no streams are active before triggering rebuild","Use a cluster-wide lock (e.g. in your orchestrator) so rebuild is never invoked concurrently","Set an alert on the isRebuilding JMX attribute"],"tags":["state","concurrency","streaming","rebuild"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}