dgraph-io/dgraph · error

while calling MovePredicate

Error message

while calling MovePredicate

What it means

movePredicate issues an internal wc.MovePredicate gRPC call to the source group's Alpha leader, telling it to stream all data for the predicate to the destination group (blocking at the leased TxnTs). If that RPC returns an error (stream failure, destination group not ready, timeout via the move context, Alpha-side processing error), the original Alpha error is wrapped with 'while calling MovePredicate'.

Source

Thrown at dgraph/cmd/zero/tablet.go:209

		return errors.Wrapf(err, "while leasing txn timestamp. Id: %+v", ids)
	}

	// Get connection to leader of source group.
	pl := s.Leader(srcGroup)
	if pl == nil {
		return errors.Errorf("No healthy connection found to leader of group %d", srcGroup)
	}
	wc := pb.NewWorkerClient(pl.Get())
	in := &pb.MovePredicatePayload{
		Predicate: predicate,
		SourceGid: srcGroup,
		DestGid:   dstGroup,
		TxnTs:     ids.StartId,
	}
	span.AddEvent(fmt.Sprintf("Move Predicate payload: %+v", in))
	glog.Infof("Starting move: %+v", in)
	if _, err := wc.MovePredicate(ctx, in); err != nil {
		return errors.Wrapf(err, "while calling MovePredicate")
	}

	p := &pb.ZeroProposal{}
	p.Tablet = &pb.Tablet{
		GroupId:           dstGroup,
		Predicate:         predicate,
		OnDiskBytes:       tab.OnDiskBytes,
		UncompressedBytes: tab.UncompressedBytes,
		Force:             true,
		MoveTs:            in.TxnTs,
	}
	msg = fmt.Sprintf("Move at Alpha done. Now proposing: %+v", p)
	span.AddEvent(fmt.Sprintf("Zero proposal: %+v", p))
	glog.Info(msg)
	if err := s.Node.proposeAndWait(ctx, p); err != nil {
		return errors.Wrapf(err, "while proposing tablet reassignment. Proposal: %+v", p)
	}
	msg = fmt.Sprintf("Predicate move done for: [%v] from group %d to %d\n",

View on GitHub (pinned to 759e242be6)

Solutions

1. Inspect the wrapped inner error in the Zero/Alpha logs for the root cause
2. Increase moveTimeout (move_timeout Zero flag) for large predicates and retry
3. Verify both source and destination Alpha groups have healthy leaders and free disk space
4. Retry the move — partial streams are rolled back via the tablet's MoveTs and are safe to redo
5. Improve network path/throughput between source and destination groups for big moves

Example fix

// before
# move of 300GB predicate times out with default timeout
// after
zero --move_timeout=4h ...   # then retry the move
curl 'localhost:6080/moveTablet?tablet=big_pred&dst_group=2'

# check both groups have leaders and destination has disk headroom
curl -s localhost:6080/state | jq -e \
  '.groups | has("1") and has("2")' \
  && df -h /path/to/dst-data | awk 'NR==2 {exit ($5+0 < 90) ? 0 : 1}' \
  || { echo "destination group not ready or disk >90%"; exit 1; }

for i in 1 2 3; do
  resp=$(curl -s "localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST")
  [[ "$(echo "$resp" | jq -r .msg)" == *"while calling MovePredicate"* ]] && { sleep 30; continue; }
  break
done

When it happens

Trigger: Source Alpha cannot reach the destination group's leader; streaming times out because the moveTimeout context expires during a large predicate copy; destination group rejects the snapshot (e.g. its leader is down or disk full); network interruption mid-stream; Alpha-side error reading Badger or exporting the predicate.

Common situations: Moving very large predicates (hundreds of GB) with the default move timeout; destination group with insufficient disk space; cross-AZ/cross-region moves with slow or unstable links; destination Alpha leader failure mid-move; retries stacking with the rebalancer re-picking the same tablet.

Related errors

• while leasing txn timestamp. Id: %+v
• No healthy connection found to leader of group %d
• recv final downstream(%d): %w
• Unhealthy connection to %v
• group: [%d] is not a known group

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/d826ef9ae1ab0bae. Report an issue: GitHub.