{"record":{"id":"8b2f9c2eef5b6faa","repo":"dgraph-io/dgraph","slug":"no-healthy-connection-found-to-leader-of-group-d","errorCode":null,"errorMessage":"No healthy connection found to leader of group %d","messagePattern":"No healthy connection found to leader of group (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"dgraph/cmd/zero/tablet.go","lineNumber":197,"sourceCode":"\tglog.Info(msg)\n\tspan.SetAttributes(attribute.String(\"tablet\", predicate))\n\tspan.SetStatus(1, msg)\n\n\t// Block all commits on this predicate. Keep them blocked until we return from this function.\n\tunblock := s.blockTablet(predicate)\n\tdefer unblock()\n\n\t// Get a new timestamp, beyond which we are sure that no new txns would be committed for this\n\t// predicate. Source Alpha leader must reach this timestamp before streaming the data.\n\tids, err := s.Timestamps(ctx, &pb.Num{Val: 1})\n\tif err != nil || ids.StartId == 0 {\n\t\treturn errors.Wrapf(err, \"while leasing txn timestamp. Id: %+v\", ids)\n\t}\n\n\t// Get connection to leader of source group.\n\tpl := s.Leader(srcGroup)\n\tif pl == nil {\n\t\treturn errors.Errorf(\"No healthy connection found to leader of group %d\", srcGroup)\n\t}\n\twc := pb.NewWorkerClient(pl.Get())\n\tin := &pb.MovePredicatePayload{\n\t\tPredicate: predicate,\n\t\tSourceGid: srcGroup,\n\t\tDestGid:   dstGroup,\n\t\tTxnTs:     ids.StartId,\n\t}\n\tspan.AddEvent(fmt.Sprintf(\"Move Predicate payload: %+v\", in))\n\tglog.Infof(\"Starting move: %+v\", in)\n\tif _, err := wc.MovePredicate(ctx, in); err != nil {\n\t\treturn errors.Wrapf(err, \"while calling MovePredicate\")\n\t}\n\n\tp := &pb.ZeroProposal{}\n\tp.Tablet = &pb.Tablet{\n\t\tGroupId:           dstGroup,\n\t\tPredicate:         predicate,","sourceCodeStart":179,"sourceCodeEnd":215,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/dgraph/cmd/zero/tablet.go#L179-L215","documentation":"After leasing the timestamp, movePredicate needs a healthy client connection to the leader of the SOURCE Alpha group (s.Leader(srcGroup)) to issue the internal MovePredicate stream RPC. If Zero has no healthy connection to that group's leader (leader unknown, all Alphas of the group unreachable, or connection pool empty), it aborts with this error.","triggerScenarios":"Source group's Alpha leader is down or restarting; all Alphas of the source group are unreachable from Zero (network/firewall on port 5080); Zero's membership map has no healthy leader entry for the group (leader election in progress in the Alpha raft group); source group has zero healthy members.","commonSituations":"Moving a predicate away from a group whose leader just crashed (the exact scenario that prompts manual moves); Kubernetes pod restarts removing the Alpha leader; security groups blocking Zero→Alpha gRPC (internal port 5080); Alpha group stuck without a leader after a quorum loss.","solutions":["Check /state for the source group's leader address and /health for the Alphas","Restore connectivity from Zero to the source Alphas on internal port 5080","Wait for the source group's Alpha raft to elect a leader, then retry","Restart or replace unhealthy Alphas in the source group so a leader exists"],"exampleFix":"// before\ncurl 'localhost:6080/moveTablet?tablet=name&dst_group=2'  # group 1 has no leader\n// after\ncurl -s localhost:6080/state | jq '.groups[\"1\"].members'   # confirm leader present\ncurl -s localhost:8080/health                               # alphas healthy\n# once group 1 has a healthy leader:\ncurl 'localhost:6080/moveTablet?tablet=name&dst_group=2'","handlingStrategy":"retry","validationCode":"# confirm source group has a reachable leader before moving\ncurl -s localhost:6080/state | jq -e --argjson g \"$SRC_GROUP\" '.groups[$g].leader // empty' \\\n  || { echo \"source group has no leader\"; exit 1; }","typeGuard":null,"tryCatchPattern":"for i in 1 2 3 4 5; do\n  resp=$(curl -s \"localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST\")\n  [[ \"$(echo \"$resp\" | jq -r .msg)\" == *\"No healthy connection\"* ]] && { sleep 20; continue; }\n  break\ndone","preventionTips":["Keep Zero→Alpha internal port 5080 connectivity open (firewalls, k8s network policies)","Monitor Alpha group leadership; alert when a group has no healthy leader","Replace crashed Alpha leaders before rebalancing away from their group"],"tags":["dgraph","zero","alpha","leader","connection"],"backgroundTag":"no-healthy-connection-to-leader","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}