dgraph-io/dgraph · error

unable to reach quorum

Error message

unable to reach quorum

What it means

Before streaming data, movePredicate calls s.latestMembershipState(ctx) to force this Zero to sync with the rest of the Zero quorum via a blocking ReadIndex/raft step. If that fails (quorum unreachable, network partition, raft timeouts), the error is wrapped with 'unable to reach quorum'. Zero cannot safely move a predicate without confirming it has an up-to-date view of membership.

Source

Thrown at dgraph/cmd/zero/tablet.go:171

	}

	// Feed the outcome of this attempt back to the rebalancer, so it stops re-picking a tablet
	// whose moves keep failing.
	start := time.Now()
	defer func() {
		s.recordMoveResult(predicate, time.Since(start), err)
	}()

	timeout := moveTimeout(predicateMoveTimeout, tab)
	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()

	span := trace.SpanFromContext(ctx)
	defer span.End()

	// Ensure that I'm connected to the rest of the Zero group, and am the leader.
	if _, err := s.latestMembershipState(ctx); err != nil {
		return errors.Wrapf(err, "unable to reach quorum")
	}
	if !s.Node.AmLeader() {
		return errors.Errorf("I am not the Zero leader")
	}
	msg := fmt.Sprintf("Going to move predicate: [%v], size: [ondisk: %v, uncompressed: %v]"+
		" from group %d to %d, timeout: %v\n", predicate, humanize.IBytes(uint64(tab.OnDiskBytes)),
		humanize.IBytes(uint64(tab.UncompressedBytes)), srcGroup, dstGroup, timeout)
	glog.Info(msg)
	span.SetAttributes(attribute.String("tablet", predicate))
	span.SetStatus(1, msg)

	// Block all commits on this predicate. Keep them blocked until we return from this function.
	unblock := s.blockTablet(predicate)
	defer unblock()

	// Get a new timestamp, beyond which we are sure that no new txns would be committed for this
	// predicate. Source Alpha leader must reach this timestamp before streaming the data.
	ids, err := s.Timestamps(ctx, &pb.Num{Val: 1})

View on GitHub (pinned to 759e242be6)

Solutions

1. Check all Zero replicas are up: /health and /state on each Zero
2. Restore network connectivity on Zero internal port 5080 (and 6080) between Zero nodes
3. Wait for Zero quorum/leader election to complete before retrying moves
4. Scale Zero back to a majority (restart the down replicas)
5. Retry the move once the cluster is healthy — the operation is safe to retry

Example fix

// before
# issuing move while zero-2 and zero-3 are down
// after
curl -s localhost:6080/health   # verify all zeros healthy first
docker start zero-2 zero-3      # restore quorum
curl 'localhost:6080/moveTablet?tablet=name&dst_group=2'

# verify all Zero replicas are healthy before moving
for z in zero1:6080 zero2:6080 zero3:6080; do
  curl -sf "$z/health" | jq -e '.[0].status == "healthy"' || { echo "$z unhealthy"; exit 1; }
done

for i in 1 2 3 4 5; do
  resp=$(curl -s "localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST")
  [[ "$(echo "$resp" | jq -r .msg)" == *"unable to reach quorum"* ]] && { sleep 15; continue; }
  break
done

When it happens

Trigger: Fewer than a majority of Zero nodes reachable (1-node Zero down/restarting, or 3-node Zero with 2 down); network partition between Zero nodes; Zero raft still electing a leader after restart; heavy load causing raft timeouts.

Common situations: Operating a multi-node Zero cluster where one replica is down for maintenance; firewall/security-group changes blocking Zero-to-Zero ports (5080 internal, 6080 gRPC); restarting the whole cluster and issuing moves before Zero converges; disk saturation on a Zero node slowing raft.

Related errors

• Unhealthy connection to %v
• I am not the Zero leader
• while leasing txn timestamp. Id: %+v
• group: [%d] is not a known group
• namespace: %d. No tablet found for: %s

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/be9a648854421499. Report an issue: GitHub.