dgraph-io/dgraph · critical

cannot propose restore request

Error message

cannot propose restore request

What it means

Wrapped by grpcWorker.Restore at worker/online_restore.go:192. After waiting on the oracle, the Alpha proposes the Restore request through its Raft group via groups().Node.proposeAndWait. This error wraps whatever errRestoreProposal/raft-level failure the proposal returned, so the restore was not applied by the group.

Source

Thrown at worker/online_restore.go:192

}

// Restore implements the Worker interface.
func (w *grpcWorker) Restore(ctx context.Context, req *pb.RestoreRequest) (*pb.Status, error) {
	var emptyRes pb.Status
	if !groups().ServesGroup(req.GroupId) {
		return &emptyRes, errors.Errorf("this server doesn't serve group id: %v", req.GroupId)
	}

	// We should wait to ensure that we have seen all the updates until the StartTs
	// of this restore transaction.
	if err := posting.Oracle().WaitForTs(ctx, req.RestoreTs); err != nil {
		return nil, errors.Wrapf(err, "cannot wait for restore ts %d", req.RestoreTs)
	}

	glog.Infof("Proposing restore request")
	err := groups().Node.proposeAndWait(ctx, &pb.Proposal{Restore: req})
	if err != nil {
		return &emptyRes, errors.Wrapf(err, errRestoreProposal)
	}

	return &emptyRes, nil
}

// TODO(DGRAPH-1232): Ensure all groups receive the restore proposal.
func handleRestoreProposal(ctx context.Context, req *pb.RestoreRequest, pidx uint64) error {
	if req == nil {
		return errors.Errorf("nil restore request")
	}

	// This is a minor inconvenience while using the incremental restore API that
	// when incrementalFrom is set to 1, we throw an error back. The restore API
	// takes two backup numbers incrementalFrom & backupNum and restores all the
	// backups including both the ends, i.e. following set notation all the backups
	// in the set [incrementalFrom, backupNum] are restored. This should work fine
	// when incrementalFrom is set to 1 which is a full backup.
	if req.IncrementalFrom == 1 {

View on GitHub (pinned to 759e242be6)

Solutions

1. Check that the group's Alpha is the leader and that all Alphas of the group are healthy (curl /state).
2. Wait for leader election to settle and retry the restore.
3. Ensure the whole cluster is up (all Zero nodes and all Alpha replicas) before restoring.
4. Investigate the underlying wrapped raft error in Alpha logs to find the exact proposal failure.

Example fix

// before
curl -X POST localhost:8080/restore -d '{...}'   # during leader change
// after
# wait until /state shows a stable leader for the group, then retry
curl localhost:8080/state | jq '.groups[].leader'
curl -X POST localhost:8080/restore -d '{...}'

// Before restore, verify a stable leader for the group
st, _ := http.Get("http://alpha:8080/state")
// assert .groups[i].leader is set and all members healthy

err := restore(ctx, req)
if err != nil && strings.Contains(err.Error(), "cannot propose restore") {
    // check leader election/quorum, wait for cluster stability, then retry
    time.Sleep(30 * time.Second)
    err = restore(ctx, req)
}

When it happens

Trigger: proposeAndWait fails because the Alpha lost leadership mid-proposal, the Raft group has no quorum, another concurrent proposal (e.g. a drop or moveTablet) conflicts, or the proposal timed out waiting for a majority.

Common situations: Restoring during a Zero/Alpha leader election; cluster with an unhealthy or down Alpha so the group has no majority; two restore/drop operations racing; network partition between Alpha and Zero.

Related errors

• failed to get connection pool for group %d
• nil restore request
• Server is not leader of this group
• ReceivePredicate failed: Not the leader of group
• unable to reach quorum

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/1e6f34a3c97e2b7e. Report an issue: GitHub.