What causes "Could not finish join for during replacement" in apache/cassandra?

Run `nodetool bootstrap resume` to retry the failed streaming and resume the join.

apache/cassandra · error · IllegalStateException

Could not finish join for during replacement

Error message

Could not finish join for during replacement

What it means

Thrown by ReplaceSameAddress.streamData when streaming of data from the replaced node fails, so the replacement cannot complete its join into the ring. Cassandra aborts the join because proceeding without the streamed data would leave the new node with incomplete replica data. It instructs the operator to inspect bootstrap state with nodetool and resume the operation.

Source

Thrown at src/java/org/apache/cassandra/tcm/sequences/ReplaceSameAddress.java:89

        BootstrapAndReplace.gossipStateToHibernate(metadata, nodeId);

        SystemKeyspace.updateLocalTokens(metadata.tokenMap.tokens(nodeId));

        if (shouldBootstrap)
        {
            boolean dataAvailable = BootstrapAndJoin.bootstrap(metadata.tokenMap.tokens(nodeId),
                                                               StorageService.INDEFINITE,
                                                               metadata,
                                                               metadata.directory.endpoint(nodeId),
                                                               movementMap(nodeId, metadata.placements(), metadata.directory),
                                                               null);

            if (!dataAvailable)
            {
                logger.warn("Some data streaming failed. Use nodetool to check bootstrap state and resume. " +
                            "For more, see `nodetool help bootstrap`. {}", SystemKeyspace.getBootstrapState());
                throw new IllegalStateException("Could not finish join for during replacement");
            }
        }

        if (finishJoiningRing)
        {
            SystemKeyspace.setBootstrapState(SystemKeyspace.BootstrapState.COMPLETED);
            StreamSupport.stream(ColumnFamilyStore.all().spliterator(), false)
                         .filter(cfs -> Schema.instance.getUserKeyspaces().names().contains(cfs.keyspace.getName()))
                         .forEach(cfs -> cfs.indexManager.executePreJoinTasksBlocking(true));
            BootstrapAndReplace.gossipStateToNormal(metadata, metadata.myNodeId());
            Gossiper.instance.mergeNodeToGossip(metadata.myNodeId(), metadata);

            // this node might have just bootstrapped; check if we should run repair immediately
            AutoRepairUtils.runRepairOnNewlyBootstrappedNodeIfEnabled();
        }
    }
}

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Run `nodetool bootstrap resume` to retry the failed streaming and resume the join.
Check `nodetool netstats` and `nodetool bootstrap` to see which streams failed and from which peers.
Verify network connectivity/firewalls between the replacement node and all stream sources, then restart the node to re-attempt.
If the node is unusable, wipe its data and restart the replacement procedure from scratch.

Example fix

// before
throw new IllegalStateException("Could not finish join for during replacement");
// after
// resume instead of leaving the node half-joined:
//   nodetool bootstrap resume
// (operator action; in code, catch and surface the failed peers)
List<InetAddressAndPort> failed = streamPlans.stream()
    .flatMap(p -> p.getFailedPeers().stream())
    .collect(Collectors.toList());
throw new IllegalStateException("Could not finish join during replacement; failed stream peers: " + failed);

Defensive patterns

Strategy: retry

Validate before calling

// before starting replacement, verify connectivity to stream sources
for (InetAddressAndPort peer : replicaPeers) {
    if (!StreamManager.instance().isReachable(peer))
        throw new RuntimeException("Cannot reach stream source " + peer + "; fix network before replacing");
}

Try / catch

try {
    replaceSameAddress.join();
} catch (IllegalStateException e) {
    logger.warn("Replacement join incomplete: {} — run `nodetool bootstrap resume`", e.getMessage());
    // do NOT wipe data; resume is the supported recovery path
    waitForOperatorOrScheduleResume();
}

Prevention

Verify network/firewall paths to all replica peers before starting a replacement.
Monitor `nodetool netstats` during streaming to catch failures early.
Avoid replacing nodes during peak load; streaming timeouts are more likely.
Keep the replacement node's data directories intact so `nodetool bootstrap resume` can recover.
Check source-node logs for stream session errors when the join fails.

When it happens

Trigger: Calling the replace-same-address bootstrap path (ReplaceSameAddress) while a data stream from any live replica of the ranges to fetch fails (network partition, dropped stream session, source node down).

Common situations: Replacing a dead node with a new one at the same IP during rack or hardware maintenance; streaming sources crash mid-transfer; firewall or MTU issues break streaming connections; cluster under heavy load causes stream session timeouts.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/e04ec65baed794f0. Report an issue: GitHub.