dgraph-io/dgraph · error

cannot wait for restore ts %d

Error message

cannot wait for restore ts %d

What it means

This error is wrapped by grpcWorker.Restore in worker/online_restore.go:186. Before proposing a restore, the Alpha waits for its posting oracle to have seen all transactions up to req.RestoreTs (posting.Oracle().WaitForTs). The wait only fails when the passed context is done — cancelled or deadline exceeded — so this error means the restore call's context ended before the Alpha caught up to RestoreTs.

Source

Thrown at worker/online_restore.go:186

	}
	con := pl.Get()
	c := pb.NewWorkerClient(con)

	_, err := c.Restore(ctx, req)
	return err
}

// Restore implements the Worker interface.
func (w *grpcWorker) Restore(ctx context.Context, req *pb.RestoreRequest) (*pb.Status, error) {
	var emptyRes pb.Status
	if !groups().ServesGroup(req.GroupId) {
		return &emptyRes, errors.Errorf("this server doesn't serve group id: %v", req.GroupId)
	}

	// We should wait to ensure that we have seen all the updates until the StartTs
	// of this restore transaction.
	if err := posting.Oracle().WaitForTs(ctx, req.RestoreTs); err != nil {
		return nil, errors.Wrapf(err, "cannot wait for restore ts %d", req.RestoreTs)
	}

	glog.Infof("Proposing restore request")
	err := groups().Node.proposeAndWait(ctx, &pb.Proposal{Restore: req})
	if err != nil {
		return &emptyRes, errors.Wrapf(err, errRestoreProposal)
	}

	return &emptyRes, nil
}

// TODO(DGRAPH-1232): Ensure all groups receive the restore proposal.
func handleRestoreProposal(ctx context.Context, req *pb.RestoreRequest, pidx uint64) error {
	if req == nil {
		return errors.Errorf("nil restore request")
	}

	// This is a minor inconvenience while using the incremental restore API that

View on GitHub (pinned to 759e242be6)

Solutions

1. Increase the context deadline/client timeout for the restore call (restore of large backups can take minutes to hours).
2. Check Alpha health and mutation backlog (waitForTs watermark via /state) and retry once the cluster is caught up.
3. Retry the restore with a fresh long-lived context; WaitForTs is safe to re-enter.
4. If the oracle never advances, investigate Alpha's raft/posting apply pipeline for stuck transactions before retrying.

Example fix

// before
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
_, err := worker.Restore(ctx, req)
// after
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)
_, err := worker.Restore(ctx, req)

// Pre-check oracle readiness before issuing restore (best-effort)
resp, _ := http.Get("http://alpha:8080/state")
// confirm Alpha healthy; then use a generous timeout
ctx, cancel := context.WithTimeout(context.Background(), time.Hour)
defer cancel()

ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)
defer cancel()
_, err := restore(ctx, req)
if err != nil && strings.Contains(err.Error(), "cannot wait for restore ts") {
    // context deadline exceeded while waiting: retry with longer deadline
    // or check Alpha caught-up watermark before retrying
}

When it happens

Trigger: Calling /restore (Admin gRPC or HTTP) when the client context deadline is shorter than the time the Alpha needs for its oracle MaxAssigned watermark to reach RestoreTs; cancelling the restore request mid-flight; an Alpha that is far behind on applying mutations so WaitForTs blocks until ctx.Done().

Common situations: HTTP proxy/ingress timeouts shorter than restore duration; gql client with a fixed timeout (e.g. 30s) issued against a busy cluster; k8s liveness/probe cancellation; operator pressing Ctrl-C on a slow restore right after sending it.

Related errors

• while leasing txn timestamp. Id: %+v
• while calling MovePredicate
• another restore operation is already running
• Pending transactions found. Please retry operation
• while retrieving manifests

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/f2ea9edb63c00ef8. Report an issue: GitHub.