weaviate/weaviate · error

failed to update replica status to '%s': %w

Error message

failed to update replica status to '%s': %w

What it means

The consumer could not report the new replication op state to the leader via leaderClient.ReplicationUpdateReplicaOpStatus. The error is returned (retryable) so backoff retries the whole state transition; repeated failure eventually fails the op.

Source

Thrown at cluster/replication/consumer.go:465

			}
			// A cancellation divert is not a failure: skip the error budget.
			if errors.Is(err, errOpCancelled) {
				return api.ShardReplicationState(""), backoff.Permanent(err)
			}
			logger.Warnf("state transition handler failed: %v", err)
			// Otherwise, register the error with the FSM
			if err := c.leaderClient.ReplicationRegisterError(ctx, op.Op.ID, err.Error()); err != nil {
				logger.Errorf("failed to register error for replication operation: %v", err)
			}
			return api.ShardReplicationState(""), err
		}

		if err := c.checkCancelled(logger, op); err != nil {
			return api.ShardReplicationState(""), backoff.Permanent(fmt.Errorf("error while checking if op is cancelled: %w", err))
		}
		if err := c.leaderClient.ReplicationUpdateReplicaOpStatus(ctx, op.Op.ID, nextState); err != nil {
			logger.WithError(err).Errorf("failed to update replica status to '%s'", nextState)
			return api.ShardReplicationState(""), fmt.Errorf("failed to update replica status to '%s': %w", nextState, err)
		}
		return nextState, nil
	}, c.backoffPolicy)
	if err != nil {
		return err
	}

	if nextState == DELETED {
		// Stop the recursion if we are in the DELETED state and don't update the state in the FSM
		return nil
	}

	op.Status.ChangeState(nextState)
	if nextState == api.READY {
		// No need to continue the recursion if we are in the READY state
		return nil
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Verify leader availability and connectivity: the consumer retries with backoff, so transient leader loss self-heals.
2. Check the leader's logs for ReplicationUpdateReplicaOpStatus failures.
3. If persistent, confirm cluster membership and RAFT health; the op may need re-registration after leader loss.

Example fix

// before
if err := c.leaderClient.ReplicationUpdateReplicaOpStatus(ctx, op.Op.ID, nextState); err != nil {
	logger.WithError(err).Errorf("failed to update replica status to '%s'", nextState)
	return api.ShardReplicationState(""), fmt.Errorf("failed to update replica status to '%s': %w", nextState, err)
}
// after (caller-side: use a context with timeout and retry around the whole op processing)
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
err := processStateAndTransition(ctx, op, stateFunc)

// pre-check leader reachability before issuing replication ops
if !isLeaderReachable(ctx, leaderAddr) { return fmt.Errorf("leader unreachable") }

if strings.Contains(err.Error(), "failed to update replica status") {
	// retry with backoff; the consumer already retries internally
	time.Sleep(backoff)
	return refreshOpStatus(ctx, opID)
}

When it happens

Trigger: Network partition to the leader node, leader change/loss, leader gRPC server down, or timeout while pushing the status update for op ID.

Common situations: Rolling restarts of the cluster where the leader is temporarily unavailable; RAFT leadership election in progress; network flakiness between replicas.

Related errors

• get gRPC connection to %s: %w
• gRPC PutObject: %w
• gRPC Abort: %w
• gRPC FetchObject: %w
• gRPC FetchObjects: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/8146ff46b5825679. Report an issue: GitHub.