{"record":{"id":"d826ef9ae1ab0bae","repo":"dgraph-io/dgraph","slug":"while-calling-movepredicate","errorCode":null,"errorMessage":"while calling MovePredicate","messagePattern":"while calling MovePredicate","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"dgraph/cmd/zero/tablet.go","lineNumber":209,"sourceCode":"\t\treturn errors.Wrapf(err, \"while leasing txn timestamp. Id: %+v\", ids)\n\t}\n\n\t// Get connection to leader of source group.\n\tpl := s.Leader(srcGroup)\n\tif pl == nil {\n\t\treturn errors.Errorf(\"No healthy connection found to leader of group %d\", srcGroup)\n\t}\n\twc := pb.NewWorkerClient(pl.Get())\n\tin := &pb.MovePredicatePayload{\n\t\tPredicate: predicate,\n\t\tSourceGid: srcGroup,\n\t\tDestGid:   dstGroup,\n\t\tTxnTs:     ids.StartId,\n\t}\n\tspan.AddEvent(fmt.Sprintf(\"Move Predicate payload: %+v\", in))\n\tglog.Infof(\"Starting move: %+v\", in)\n\tif _, err := wc.MovePredicate(ctx, in); err != nil {\n\t\treturn errors.Wrapf(err, \"while calling MovePredicate\")\n\t}\n\n\tp := &pb.ZeroProposal{}\n\tp.Tablet = &pb.Tablet{\n\t\tGroupId:           dstGroup,\n\t\tPredicate:         predicate,\n\t\tOnDiskBytes:       tab.OnDiskBytes,\n\t\tUncompressedBytes: tab.UncompressedBytes,\n\t\tForce:             true,\n\t\tMoveTs:            in.TxnTs,\n\t}\n\tmsg = fmt.Sprintf(\"Move at Alpha done. Now proposing: %+v\", p)\n\tspan.AddEvent(fmt.Sprintf(\"Zero proposal: %+v\", p))\n\tglog.Info(msg)\n\tif err := s.Node.proposeAndWait(ctx, p); err != nil {\n\t\treturn errors.Wrapf(err, \"while proposing tablet reassignment. Proposal: %+v\", p)\n\t}\n\tmsg = fmt.Sprintf(\"Predicate move done for: [%v] from group %d to %d\\n\",","sourceCodeStart":191,"sourceCodeEnd":227,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/dgraph/cmd/zero/tablet.go#L191-L227","documentation":"movePredicate issues an internal wc.MovePredicate gRPC call to the source group's Alpha leader, telling it to stream all data for the predicate to the destination group (blocking at the leased TxnTs). If that RPC returns an error (stream failure, destination group not ready, timeout via the move context, Alpha-side processing error), the original Alpha error is wrapped with 'while calling MovePredicate'.","triggerScenarios":"Source Alpha cannot reach the destination group's leader; streaming times out because the moveTimeout context expires during a large predicate copy; destination group rejects the snapshot (e.g. its leader is down or disk full); network interruption mid-stream; Alpha-side error reading Badger or exporting the predicate.","commonSituations":"Moving very large predicates (hundreds of GB) with the default move timeout; destination group with insufficient disk space; cross-AZ/cross-region moves with slow or unstable links; destination Alpha leader failure mid-move; retries stacking with the rebalancer re-picking the same tablet.","solutions":["Inspect the wrapped inner error in the Zero/Alpha logs for the root cause","Increase moveTimeout (move_timeout Zero flag) for large predicates and retry","Verify both source and destination Alpha groups have healthy leaders and free disk space","Retry the move — partial streams are rolled back via the tablet's MoveTs and are safe to redo","Improve network path/throughput between source and destination groups for big moves"],"exampleFix":"// before\n# move of 300GB predicate times out with default timeout\n// after\nzero --move_timeout=4h ...   # then retry the move\ncurl 'localhost:6080/moveTablet?tablet=big_pred&dst_group=2'","handlingStrategy":"retry","validationCode":"# check both groups have leaders and destination has disk headroom\ncurl -s localhost:6080/state | jq -e \\\n  '.groups | has(\"1\") and has(\"2\")' \\\n  && df -h /path/to/dst-data | awk 'NR==2 {exit ($5+0 < 90) ? 0 : 1}' \\\n  || { echo \"destination group not ready or disk >90%\"; exit 1; }","typeGuard":null,"tryCatchPattern":"for i in 1 2 3; do\n  resp=$(curl -s \"localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST\")\n  [[ \"$(echo \"$resp\" | jq -r .msg)\" == *\"while calling MovePredicate\"* ]] && { sleep 30; continue; }\n  break\ndone","preventionTips":["Increase --move_timeout on Zero for large predicates","Ensure destination Alphas have ample free disk before moves","Verify network bandwidth between source and destination groups (prefer same AZ/rack)","Read the inner wrapped error in Zero/Alpha logs — this error only wraps the real cause","Avoid running the rebalancer and manual moves of the same tablet simultaneously"],"tags":["dgraph","zero","alpha","grpc","streaming","timeout"],"backgroundTag":"movepredicate-rpc-failed","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}