{"record":{"id":"8146ff46b5825679","repo":"weaviate/weaviate","slug":"failed-to-update-replica-status-to-s-w","errorCode":null,"errorMessage":"failed to update replica status to '%s': %w","messagePattern":"failed to update replica status to '(.+?)': %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cluster/replication/consumer.go","lineNumber":465,"sourceCode":"\t\t\t}\n\t\t\t// A cancellation divert is not a failure: skip the error budget.\n\t\t\tif errors.Is(err, errOpCancelled) {\n\t\t\t\treturn api.ShardReplicationState(\"\"), backoff.Permanent(err)\n\t\t\t}\n\t\t\tlogger.Warnf(\"state transition handler failed: %v\", err)\n\t\t\t// Otherwise, register the error with the FSM\n\t\t\tif err := c.leaderClient.ReplicationRegisterError(ctx, op.Op.ID, err.Error()); err != nil {\n\t\t\t\tlogger.Errorf(\"failed to register error for replication operation: %v\", err)\n\t\t\t}\n\t\t\treturn api.ShardReplicationState(\"\"), err\n\t\t}\n\n\t\tif err := c.checkCancelled(logger, op); err != nil {\n\t\t\treturn api.ShardReplicationState(\"\"), backoff.Permanent(fmt.Errorf(\"error while checking if op is cancelled: %w\", err))\n\t\t}\n\t\tif err := c.leaderClient.ReplicationUpdateReplicaOpStatus(ctx, op.Op.ID, nextState); err != nil {\n\t\t\tlogger.WithError(err).Errorf(\"failed to update replica status to '%s'\", nextState)\n\t\t\treturn api.ShardReplicationState(\"\"), fmt.Errorf(\"failed to update replica status to '%s': %w\", nextState, err)\n\t\t}\n\t\treturn nextState, nil\n\t}, c.backoffPolicy)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tif nextState == DELETED {\n\t\t// Stop the recursion if we are in the DELETED state and don't update the state in the FSM\n\t\treturn nil\n\t}\n\n\top.Status.ChangeState(nextState)\n\tif nextState == api.READY {\n\t\t// No need to continue the recursion if we are in the READY state\n\t\treturn nil\n\t}\n","sourceCodeStart":447,"sourceCodeEnd":483,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/cluster/replication/consumer.go#L447-L483","documentation":"The consumer could not report the new replication op state to the leader via leaderClient.ReplicationUpdateReplicaOpStatus. The error is returned (retryable) so backoff retries the whole state transition; repeated failure eventually fails the op.","triggerScenarios":"Network partition to the leader node, leader change/loss, leader gRPC server down, or timeout while pushing the status update for op ID.","commonSituations":"Rolling restarts of the cluster where the leader is temporarily unavailable; RAFT leadership election in progress; network flakiness between replicas.","solutions":["Verify leader availability and connectivity: the consumer retries with backoff, so transient leader loss self-heals.","Check the leader's logs for ReplicationUpdateReplicaOpStatus failures.","If persistent, confirm cluster membership and RAFT health; the op may need re-registration after leader loss."],"exampleFix":"// before\nif err := c.leaderClient.ReplicationUpdateReplicaOpStatus(ctx, op.Op.ID, nextState); err != nil {\n\tlogger.WithError(err).Errorf(\"failed to update replica status to '%s'\", nextState)\n\treturn api.ShardReplicationState(\"\"), fmt.Errorf(\"failed to update replica status to '%s': %w\", nextState, err)\n}\n// after (caller-side: use a context with timeout and retry around the whole op processing)\nctx, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\nerr := processStateAndTransition(ctx, op, stateFunc)","handlingStrategy":"retry","validationCode":"// pre-check leader reachability before issuing replication ops\nif !isLeaderReachable(ctx, leaderAddr) { return fmt.Errorf(\"leader unreachable\") }","typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"failed to update replica status\") {\n\t// retry with backoff; the consumer already retries internally\n\ttime.Sleep(backoff)\n\treturn refreshOpStatus(ctx, opID)\n}","preventionTips":["Keep RAFT/leader healthy; alert on leader churn","Ensure inter-node network reliability for the replication port","Use reasonable RPC timeouts"],"tags":["replication","raft","network","grpc"],"backgroundTag":"leader-unavailable","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}