What causes "while leasing txn timestamp. Id: %+v" in dgraph-io/dgraph?

Retry the move — the timestamp lease is transactional and safe to redo

while leasing txn timestamp. Id: %+v

Error message

while leasing txn timestamp. Id: %+v

What it means

Before streaming, movePredicate leases one transaction timestamp from Zero's timestamp allocator (s.Timestamps). This timestamp marks the point beyond which no new commits happen for the predicate, so the source Alpha knows where to stop. If leasing fails or returns a zero StartId, the error (if any) is wrapped with 'while leasing txn timestamp'.

Source

Thrown at dgraph/cmd/zero/tablet.go:191

	if !s.Node.AmLeader() {
		return errors.Errorf("I am not the Zero leader")
	}
	msg := fmt.Sprintf("Going to move predicate: [%v], size: [ondisk: %v, uncompressed: %v]"+
		" from group %d to %d, timeout: %v\n", predicate, humanize.IBytes(uint64(tab.OnDiskBytes)),
		humanize.IBytes(uint64(tab.UncompressedBytes)), srcGroup, dstGroup, timeout)
	glog.Info(msg)
	span.SetAttributes(attribute.String("tablet", predicate))
	span.SetStatus(1, msg)

	// Block all commits on this predicate. Keep them blocked until we return from this function.
	unblock := s.blockTablet(predicate)
	defer unblock()

	// Get a new timestamp, beyond which we are sure that no new txns would be committed for this
	// predicate. Source Alpha leader must reach this timestamp before streaming the data.
	ids, err := s.Timestamps(ctx, &pb.Num{Val: 1})
	if err != nil || ids.StartId == 0 {
		return errors.Wrapf(err, "while leasing txn timestamp. Id: %+v", ids)
	}

	// Get connection to leader of source group.
	pl := s.Leader(srcGroup)
	if pl == nil {
		return errors.Errorf("No healthy connection found to leader of group %d", srcGroup)
	}
	wc := pb.NewWorkerClient(pl.Get())
	in := &pb.MovePredicatePayload{
		Predicate: predicate,
		SourceGid: srcGroup,
		DestGid:   dstGroup,
		TxnTs:     ids.StartId,
	}
	span.AddEvent(fmt.Sprintf("Move Predicate payload: %+v", in))
	glog.Infof("Starting move: %+v", in)
	if _, err := wc.MovePredicate(ctx, in); err != nil {
		return errors.Wrapf(err, "while calling MovePredicate")

View on GitHub (pinned to 759e242be6)

Solutions

Retry the move — the timestamp lease is transactional and safe to redo
Increase moveTimeout if moves of large tablets keep timing out
Verify this Zero is still leader and quorum is healthy via /state and /health
Check Zero logs for raft errors concurrent with the failed lease

Example fix

// before
# default move timeout too small for 200GB predicate
// after
# restart zero with a larger move window, then retry
zero --move_timeout=2h --my=... --replicas=...

Defensive patterns

Strategy: retry

Validate before calling

# ensure zero is leader and healthy before moving
curl -sf localhost:6080/state >/dev/null && curl -sf localhost:6080/health >/dev/null \
  || { echo "zero not healthy/leader"; exit 1; }

Try / catch

for i in 1 2 3; do
  resp=$(curl -s "localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST")
  [[ "$(echo "$resp" | jq -r .msg)" == *"leasing txn timestamp"* ]] && { sleep 10; continue; }
  break
done

Prevention

Set an adequate --move_timeout on Zero for your largest predicates
Monitor Zero raft health and leadership stability
Avoid issuing many concurrent moves that contend for Zero raft proposals

When it happens

Trigger: s.Timestamps returns an error (context timeout — moveTimeout expired; Zero raft write failing because the node lost leadership or quorum); or ids.StartId == 0 indicating an empty/invalid allocator response, typically when the proposal did not commit.

Common situations: Move timeout too small for a slow cluster (large predicate, high load); Zero lost leadership between the quorum check and the timestamp lease; Zero raft group under pressure from many concurrent proposals (other moves, membership changes).

Related errors

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/06e7586a1d63ca66. Report an issue: GitHub.