weaviate/weaviate · error

add replica %q to shard %q in collection %q: %w

Error message

add replica %q to shard %q in collection %q: %w

What it means

Wraps a failure from s.AddReplicaToShard when trying to place a replacement replica on a candidate node after a node departure. addReplacementReplicas picks new nodes not equal to the removed node and not already holding the shard, then issues the add command via Raft; failure means that command was rejected or failed.

Source

Thrown at cluster/raft_cluster_endpoints.go:249

			"removed_node": removedNode,
		}).Warnf("skipping replacement replica after node departure: %v", err)
		return nil
	}

	// Always ensure we have at least desiredRF + 1 replicas before deletion
	targetCount := desiredRF + 1
	currentCount := len(currentReplicas)

	for _, newNode := range candidates {
		if currentCount >= targetCount {
			break
		}
		if newNode == removedNode || slices.Contains(currentReplicas, newNode) {
			continue
		}

		if _, err := s.AddReplicaToShard(ctx, className, shardName, newNode); err != nil {
			return fmt.Errorf("add replica %q to shard %q in collection %q: %w", newNode, shardName, className, err)
		}

		currentCount++
	}

	return nil
}

// replicaCandidates returns the nodes eligible to hold a replica of className.
// A namespace-qualified class is pinned to its namespace's home_node, so that
// node is its only candidate.
func (s *Raft) replicaCandidates(className string) ([]string, error) {
	storageCandidates := s.StorageCandidates()

	namespace := namespacing.NamespaceFromQualified(className)
	if namespace == "" {
		return storageCandidates, nil
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Verify the candidate node is healthy, fully joined, and has capacity, then retry the removal/rebalance
2. Ensure the cluster has more healthy nodes than the replication factor so replacements can be placed
3. Check whether the replica already exists (concurrent repair may have filled the slot) and re-run rebalance to converge
4. Review Raft leader logs for the underlying rejection reason and fix the cluster-level cause

func canHostReplica(node string, currentReplicas []string, removedNode string, healthy map[string]bool) bool {
  return node != removedNode && !slices.Contains(currentReplicas, node) && healthy[node]
}

if err := raft.Remove(ctx, nodeID); err != nil {
  if strings.Contains(err.Error(), "add replica") {
    // candidate node may not have been ready — wait for cluster health, retry
    time.Sleep(5 * time.Second)
    err = raft.Remove(ctx, nodeID)
  }
}

When it happens

Trigger: Adding a replacement replica fails because the target node is unavailable, the shard/class no longer exists, RF restoration is attempted while the cluster cannot host the replica (disk full, node draining), or a concurrent operation already modified replicas.

Common situations: Decommissioning a node in a cluster at minimum capacity (RF equal to node count); target candidate node restarting or joining but not fully ready; replicated shard corruption causing the add to be refused.

Related errors

• delete replica %q from shard %q in collection %q: %w
• rebalance replicas after removing node %q: %w
• get replicas for shard %q in collection %q: %w
• shard not found
• get replicas for shard %q in collection %q before deletion:

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/16e2f36e387abe54. Report an issue: GitHub.