{"record":{"id":"d03b79750695afa0","repo":"apache/cassandra","slug":"can-not-decommission-a-node-that-has-an-in-progres","errorCode":null,"errorMessage":"Can not decommission a node that has an in-progress sequence","messagePattern":"Can not decommission a node that has an in-progress sequence","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/sequences/SingleNodeSequences.java","lineNumber":103,"sourceCode":"            logger.info(\"starting decommission with {} {}\", metadata.epoch, self);\n            // We reset transferred ranges upon starting a decommission so that we fully stream\n            // anything written since a previous attempt which may not have been persisted to a pending endpoint\n            SystemKeyspace.resetTransferredRanges();\n            logger.info(\"done resetting transferred ranges {} {}\", metadata.epoch, self);\n            ClusterMetadataService.instance().commit(new PrepareLeave(self,\n                                                                      force,\n                                                                      ClusterMetadataService.instance().placementProvider(),\n                                                                      LeaveStreams.Kind.UNBOOTSTRAP),\n                                                     m -> m,\n                                                     failureHandler(\"PrepareLeave\", StorageService.instance::markDecommissionFailed));\n        }\n        else if (InProgressSequences.isLeave(inProgress))\n        {\n            logger.info(\"Resuming decommission @ {} (current epoch = {}): {}\", inProgress.latestModification, metadata.epoch, inProgress.status());\n        }\n        else\n        {\n            throw new IllegalArgumentException(\"Can not decommission a node that has an in-progress sequence\");\n        }\n\n        StorageService.instance.clearTransientMode();\n        InProgressSequences.finishInProgressSequences(self);\n        Gossiper.instance.unsafeBroadcastLeftStatus(FBUtilities.getBroadcastAddressAndPort(),\n                                                    tokens,\n                                                    metadata.directory.allJoinedEndpoints());\n        if (shutdownNetworking)\n            StorageService.instance.shutdownNetworking();\n    }\n\n    static void abortDecommission(String nodeId)\n    {\n        abortHelper(nodeId, MultiStepOperation.Kind.LEAVE, DECOMMISSION_FAILED);\n    }\n\n    /**\n     * Entrypoint to begin node removal process","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/sequences/SingleNodeSequences.java#L85-L121","documentation":"decommission throws IllegalArgumentException when the node's ClusterMetadata contains an in-progress multi-step sequence that is neither a bootstrap nor a leave. An unrelated in-flight sequence means running decommission concurrently would conflict with cluster metadata invariants. The existing sequence must complete or be cancelled first.","triggerScenarios":"Calling decommission while metadata.inProgressSequences for this node holds a sequence other than bootstrap/leave (e.g. a replace, move, or rebuild sequence in progress).","commonSituations":"Operator decommissions while a prior topology operation (bootstrap, replace, repair-driven sequence) is unfinished; a crashed operation left a stale in-progress sequence registered in TCM.","solutions":["Wait for the in-progress sequence to finish, then decommission.","If the sequence is actually a decommission (leave), it will be resumed automatically instead of throwing.","Cancel or complete the stuck operation via the appropriate repair/abort tooling before retrying.","Inspect ClusterMetadata inProgressSequences (logs/nodetool) to identify which sequence blocks decommission."],"exampleFix":"// before\nnodetool decommission  // while a move sequence is in progress\n// after\nMultiStepOperation<?> inProgress = ClusterMetadata.current().inProgressSequences.get(self);\nif (inProgress == null || InProgressSequences.isBootstrap(inProgress) || InProgressSequences.isLeave(inProgress)) {\n    nodetool.decommission();\n}","handlingStrategy":"validation","validationCode":"MultiStepOperation<?> inProgress =\n    ClusterMetadata.current().inProgressSequences.get(ClusterMetadata.current().myNodeId());\nif (inProgress != null\n    && !InProgressSequences.isBootstrap(inProgress)\n    && !InProgressSequences.isLeave(inProgress)) {\n    throw new RuntimeException(\"In-progress sequence \" + inProgress + \" blocks decommission\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    decommission();\n} catch (IllegalArgumentException e) {\n    if (e.getMessage().contains(\"in-progress sequence\")) {\n        waitForInProgressSequenceToFinish();\n        decommission();\n    } else throw e;\n}","preventionTips":["Check for in-progress sequences (bootstrap/leave/replace) before topology changes.","Serialize topology operations; never overlap decommission with other sequences.","Clean up stale sequences after crashes before issuing new operations.","Monitor TCM logs for unfinished multi-step operations."],"tags":["decommission","tcm","in-progress-sequence","concurrency"],"backgroundTag":"invalid-state-transition","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}