dgraph-io/dgraph · error

No healthy connection found to leader of group %d

Error message

No healthy connection found to leader of group %d

What it means

After leasing the timestamp, movePredicate needs a healthy client connection to the leader of the SOURCE Alpha group (s.Leader(srcGroup)) to issue the internal MovePredicate stream RPC. If Zero has no healthy connection to that group's leader (leader unknown, all Alphas of the group unreachable, or connection pool empty), it aborts with this error.

Source

Thrown at dgraph/cmd/zero/tablet.go:197

	glog.Info(msg)
	span.SetAttributes(attribute.String("tablet", predicate))
	span.SetStatus(1, msg)

	// Block all commits on this predicate. Keep them blocked until we return from this function.
	unblock := s.blockTablet(predicate)
	defer unblock()

	// Get a new timestamp, beyond which we are sure that no new txns would be committed for this
	// predicate. Source Alpha leader must reach this timestamp before streaming the data.
	ids, err := s.Timestamps(ctx, &pb.Num{Val: 1})
	if err != nil || ids.StartId == 0 {
		return errors.Wrapf(err, "while leasing txn timestamp. Id: %+v", ids)
	}

	// Get connection to leader of source group.
	pl := s.Leader(srcGroup)
	if pl == nil {
		return errors.Errorf("No healthy connection found to leader of group %d", srcGroup)
	}
	wc := pb.NewWorkerClient(pl.Get())
	in := &pb.MovePredicatePayload{
		Predicate: predicate,
		SourceGid: srcGroup,
		DestGid:   dstGroup,
		TxnTs:     ids.StartId,
	}
	span.AddEvent(fmt.Sprintf("Move Predicate payload: %+v", in))
	glog.Infof("Starting move: %+v", in)
	if _, err := wc.MovePredicate(ctx, in); err != nil {
		return errors.Wrapf(err, "while calling MovePredicate")
	}

	p := &pb.ZeroProposal{}
	p.Tablet = &pb.Tablet{
		GroupId:           dstGroup,
		Predicate:         predicate,

View on GitHub (pinned to 759e242be6)

Solutions

1. Check /state for the source group's leader address and /health for the Alphas
2. Restore connectivity from Zero to the source Alphas on internal port 5080
3. Wait for the source group's Alpha raft to elect a leader, then retry
4. Restart or replace unhealthy Alphas in the source group so a leader exists

Example fix

// before
curl 'localhost:6080/moveTablet?tablet=name&dst_group=2'  # group 1 has no leader
// after
curl -s localhost:6080/state | jq '.groups["1"].members'   # confirm leader present
curl -s localhost:8080/health                               # alphas healthy
# once group 1 has a healthy leader:
curl 'localhost:6080/moveTablet?tablet=name&dst_group=2'

# confirm source group has a reachable leader before moving
curl -s localhost:6080/state | jq -e --argjson g "$SRC_GROUP" '.groups[$g].leader // empty' \
  || { echo "source group has no leader"; exit 1; }

for i in 1 2 3 4 5; do
  resp=$(curl -s "localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST")
  [[ "$(echo "$resp" | jq -r .msg)" == *"No healthy connection"* ]] && { sleep 20; continue; }
  break
done

When it happens

Trigger: Source group's Alpha leader is down or restarting; all Alphas of the source group are unreachable from Zero (network/firewall on port 5080); Zero's membership map has no healthy leader entry for the group (leader election in progress in the Alpha raft group); source group has zero healthy members.

Common situations: Moving a predicate away from a group whose leader just crashed (the exact scenario that prompts manual moves); Kubernetes pod restarts removing the Alpha leader; security groups blocking Zero→Alpha gRPC (internal port 5080); Alpha group stuck without a leader after a quorum loss.

Related errors

• I am not the Zero leader
• while calling MovePredicate
• group: [%d] is not a known group
• namespace: %d. No tablet found for: %s
• namespace: %d. Tablet: [%s] is already being served by group

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/8b2f9c2eef5b6faa. Report an issue: GitHub.