{"record":{"id":"f2ea9edb63c00ef8","repo":"dgraph-io/dgraph","slug":"cannot-wait-for-restore-ts-d","errorCode":null,"errorMessage":"cannot wait for restore ts %d","messagePattern":"cannot wait for restore ts (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"worker/online_restore.go","lineNumber":186,"sourceCode":"\t}\n\tcon := pl.Get()\n\tc := pb.NewWorkerClient(con)\n\n\t_, err := c.Restore(ctx, req)\n\treturn err\n}\n\n// Restore implements the Worker interface.\nfunc (w *grpcWorker) Restore(ctx context.Context, req *pb.RestoreRequest) (*pb.Status, error) {\n\tvar emptyRes pb.Status\n\tif !groups().ServesGroup(req.GroupId) {\n\t\treturn &emptyRes, errors.Errorf(\"this server doesn't serve group id: %v\", req.GroupId)\n\t}\n\n\t// We should wait to ensure that we have seen all the updates until the StartTs\n\t// of this restore transaction.\n\tif err := posting.Oracle().WaitForTs(ctx, req.RestoreTs); err != nil {\n\t\treturn nil, errors.Wrapf(err, \"cannot wait for restore ts %d\", req.RestoreTs)\n\t}\n\n\tglog.Infof(\"Proposing restore request\")\n\terr := groups().Node.proposeAndWait(ctx, &pb.Proposal{Restore: req})\n\tif err != nil {\n\t\treturn &emptyRes, errors.Wrapf(err, errRestoreProposal)\n\t}\n\n\treturn &emptyRes, nil\n}\n\n// TODO(DGRAPH-1232): Ensure all groups receive the restore proposal.\nfunc handleRestoreProposal(ctx context.Context, req *pb.RestoreRequest, pidx uint64) error {\n\tif req == nil {\n\t\treturn errors.Errorf(\"nil restore request\")\n\t}\n\n\t// This is a minor inconvenience while using the incremental restore API that","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/worker/online_restore.go#L168-L204","documentation":"This error is wrapped by grpcWorker.Restore in worker/online_restore.go:186. Before proposing a restore, the Alpha waits for its posting oracle to have seen all transactions up to req.RestoreTs (posting.Oracle().WaitForTs). The wait only fails when the passed context is done — cancelled or deadline exceeded — so this error means the restore call's context ended before the Alpha caught up to RestoreTs.","triggerScenarios":"Calling /restore (Admin gRPC or HTTP) when the client context deadline is shorter than the time the Alpha needs for its oracle MaxAssigned watermark to reach RestoreTs; cancelling the restore request mid-flight; an Alpha that is far behind on applying mutations so WaitForTs blocks until ctx.Done().","commonSituations":"HTTP proxy/ingress timeouts shorter than restore duration; gql client with a fixed timeout (e.g. 30s) issued against a busy cluster; k8s liveness/probe cancellation; operator pressing Ctrl-C on a slow restore right after sending it.","solutions":["Increase the context deadline/client timeout for the restore call (restore of large backups can take minutes to hours).","Check Alpha health and mutation backlog (waitForTs watermark via /state) and retry once the cluster is caught up.","Retry the restore with a fresh long-lived context; WaitForTs is safe to re-enter.","If the oracle never advances, investigate Alpha's raft/posting apply pipeline for stuck transactions before retrying."],"exampleFix":"// before\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\n_, err := worker.Restore(ctx, req)\n// after\nctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)\n_, err := worker.Restore(ctx, req)","handlingStrategy":"try-catch","validationCode":"// Pre-check oracle readiness before issuing restore (best-effort)\nresp, _ := http.Get(\"http://alpha:8080/state\")\n// confirm Alpha healthy; then use a generous timeout\nctx, cancel := context.WithTimeout(context.Background(), time.Hour)\ndefer cancel()","typeGuard":null,"tryCatchPattern":"ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)\ndefer cancel()\n_, err := restore(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"cannot wait for restore ts\") {\n    // context deadline exceeded while waiting: retry with longer deadline\n    // or check Alpha caught-up watermark before retrying\n}","preventionTips":["Always use a multi-hour deadline for restore calls; restores are long-running.","Do not cancel restore requests once issued; monitor progress via Alpha logs instead.","Check Alpha mutation backlog/watermark via /state before starting a restore.","Avoid restoring on a heavily loaded or lagging cluster; pause traffic if possible."],"tags":["dgraph","restore","timeout","context-canceled"],"backgroundTag":"context-deadline-exceeded","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}