{"record":{"id":"e04ec65baed794f0","repo":"apache/cassandra","slug":"could-not-finish-join-for-during-replacement","errorCode":null,"errorMessage":"Could not finish join for during replacement","messagePattern":"Could not finish join for during replacement","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/tcm/sequences/ReplaceSameAddress.java","lineNumber":89,"sourceCode":"\n        BootstrapAndReplace.gossipStateToHibernate(metadata, nodeId);\n\n        SystemKeyspace.updateLocalTokens(metadata.tokenMap.tokens(nodeId));\n\n        if (shouldBootstrap)\n        {\n            boolean dataAvailable = BootstrapAndJoin.bootstrap(metadata.tokenMap.tokens(nodeId),\n                                                               StorageService.INDEFINITE,\n                                                               metadata,\n                                                               metadata.directory.endpoint(nodeId),\n                                                               movementMap(nodeId, metadata.placements(), metadata.directory),\n                                                               null);\n\n            if (!dataAvailable)\n            {\n                logger.warn(\"Some data streaming failed. Use nodetool to check bootstrap state and resume. \" +\n                            \"For more, see `nodetool help bootstrap`. {}\", SystemKeyspace.getBootstrapState());\n                throw new IllegalStateException(\"Could not finish join for during replacement\");\n            }\n        }\n\n        if (finishJoiningRing)\n        {\n            SystemKeyspace.setBootstrapState(SystemKeyspace.BootstrapState.COMPLETED);\n            StreamSupport.stream(ColumnFamilyStore.all().spliterator(), false)\n                         .filter(cfs -> Schema.instance.getUserKeyspaces().names().contains(cfs.keyspace.getName()))\n                         .forEach(cfs -> cfs.indexManager.executePreJoinTasksBlocking(true));\n            BootstrapAndReplace.gossipStateToNormal(metadata, metadata.myNodeId());\n            Gossiper.instance.mergeNodeToGossip(metadata.myNodeId(), metadata);\n\n            // this node might have just bootstrapped; check if we should run repair immediately\n            AutoRepairUtils.runRepairOnNewlyBootstrappedNodeIfEnabled();\n        }\n    }\n}\n","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/tcm/sequences/ReplaceSameAddress.java#L71-L107","documentation":"Thrown by ReplaceSameAddress.streamData when streaming of data from the replaced node fails, so the replacement cannot complete its join into the ring. Cassandra aborts the join because proceeding without the streamed data would leave the new node with incomplete replica data. It instructs the operator to inspect bootstrap state with nodetool and resume the operation.","triggerScenarios":"Calling the replace-same-address bootstrap path (ReplaceSameAddress) while a data stream from any live replica of the ranges to fetch fails (network partition, dropped stream session, source node down).","commonSituations":"Replacing a dead node with a new one at the same IP during rack or hardware maintenance; streaming sources crash mid-transfer; firewall or MTU issues break streaming connections; cluster under heavy load causes stream session timeouts.","solutions":["Run `nodetool bootstrap resume` to retry the failed streaming and resume the join.","Check `nodetool netstats` and `nodetool bootstrap` to see which streams failed and from which peers.","Verify network connectivity/firewalls between the replacement node and all stream sources, then restart the node to re-attempt.","If the node is unusable, wipe its data and restart the replacement procedure from scratch."],"exampleFix":"// before\nthrow new IllegalStateException(\"Could not finish join for during replacement\");\n// after\n// resume instead of leaving the node half-joined:\n//   nodetool bootstrap resume\n// (operator action; in code, catch and surface the failed peers)\nList<InetAddressAndPort> failed = streamPlans.stream()\n    .flatMap(p -> p.getFailedPeers().stream())\n    .collect(Collectors.toList());\nthrow new IllegalStateException(\"Could not finish join during replacement; failed stream peers: \" + failed);","handlingStrategy":"retry","validationCode":"// before starting replacement, verify connectivity to stream sources\nfor (InetAddressAndPort peer : replicaPeers) {\n    if (!StreamManager.instance().isReachable(peer))\n        throw new RuntimeException(\"Cannot reach stream source \" + peer + \"; fix network before replacing\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    replaceSameAddress.join();\n} catch (IllegalStateException e) {\n    logger.warn(\"Replacement join incomplete: {} — run `nodetool bootstrap resume`\", e.getMessage());\n    // do NOT wipe data; resume is the supported recovery path\n    waitForOperatorOrScheduleResume();\n}","preventionTips":["Verify network/firewall paths to all replica peers before starting a replacement.","Monitor `nodetool netstats` during streaming to catch failures early.","Avoid replacing nodes during peak load; streaming timeouts are more likely.","Keep the replacement node's data directories intact so `nodetool bootstrap resume` can recover.","Check source-node logs for stream session errors when the join fails."],"tags":["streaming","bootstrap","topology","replace-node"],"backgroundTag":"bootstrap-stream-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}