{"record":{"id":"16e2f36e387abe54","repo":"weaviate/weaviate","slug":"add-replica-q-to-shard-q-in-collection-q-w","errorCode":null,"errorMessage":"add replica %q to shard %q in collection %q: %w","messagePattern":"add replica %q to shard %q in collection %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cluster/raft_cluster_endpoints.go","lineNumber":249,"sourceCode":"\t\t\t\"removed_node\": removedNode,\n\t\t}).Warnf(\"skipping replacement replica after node departure: %v\", err)\n\t\treturn nil\n\t}\n\n\t// Always ensure we have at least desiredRF + 1 replicas before deletion\n\ttargetCount := desiredRF + 1\n\tcurrentCount := len(currentReplicas)\n\n\tfor _, newNode := range candidates {\n\t\tif currentCount >= targetCount {\n\t\t\tbreak\n\t\t}\n\t\tif newNode == removedNode || slices.Contains(currentReplicas, newNode) {\n\t\t\tcontinue\n\t\t}\n\n\t\tif _, err := s.AddReplicaToShard(ctx, className, shardName, newNode); err != nil {\n\t\t\treturn fmt.Errorf(\"add replica %q to shard %q in collection %q: %w\", newNode, shardName, className, err)\n\t\t}\n\n\t\tcurrentCount++\n\t}\n\n\treturn nil\n}\n\n// replicaCandidates returns the nodes eligible to hold a replica of className.\n// A namespace-qualified class is pinned to its namespace's home_node, so that\n// node is its only candidate.\nfunc (s *Raft) replicaCandidates(className string) ([]string, error) {\n\tstorageCandidates := s.StorageCandidates()\n\n\tnamespace := namespacing.NamespaceFromQualified(className)\n\tif namespace == \"\" {\n\t\treturn storageCandidates, nil\n\t}","sourceCodeStart":231,"sourceCodeEnd":267,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/cluster/raft_cluster_endpoints.go#L231-L267","documentation":"Wraps a failure from s.AddReplicaToShard when trying to place a replacement replica on a candidate node after a node departure. addReplacementReplicas picks new nodes not equal to the removed node and not already holding the shard, then issues the add command via Raft; failure means that command was rejected or failed.","triggerScenarios":"Adding a replacement replica fails because the target node is unavailable, the shard/class no longer exists, RF restoration is attempted while the cluster cannot host the replica (disk full, node draining), or a concurrent operation already modified replicas.","commonSituations":"Decommissioning a node in a cluster at minimum capacity (RF equal to node count); target candidate node restarting or joining but not fully ready; replicated shard corruption causing the add to be refused.","solutions":["Verify the candidate node is healthy, fully joined, and has capacity, then retry the removal/rebalance","Ensure the cluster has more healthy nodes than the replication factor so replacements can be placed","Check whether the replica already exists (concurrent repair may have filled the slot) and re-run rebalance to converge","Review Raft leader logs for the underlying rejection reason and fix the cluster-level cause"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"func canHostReplica(node string, currentReplicas []string, removedNode string, healthy map[string]bool) bool {\n  return node != removedNode && !slices.Contains(currentReplicas, node) && healthy[node]\n}","typeGuard":null,"tryCatchPattern":"if err := raft.Remove(ctx, nodeID); err != nil {\n  if strings.Contains(err.Error(), \"add replica\") {\n    // candidate node may not have been ready — wait for cluster health, retry\n    time.Sleep(5 * time.Second)\n    err = raft.Remove(ctx, nodeID)\n  }\n}","preventionTips":["Run at least rf+1 healthy nodes so replacement replicas always have a home","Verify candidate nodes are fully joined and have disk capacity before removal","Check that no concurrent operation already added the replica before retrying","Monitor node readiness/health between removal and rebalance completion"],"tags":["go","raft","replication","shard","node-removal"],"backgroundTag":"replica-addition-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}