{"record":{"id":"be9a648854421499","repo":"dgraph-io/dgraph","slug":"unable-to-reach-quorum","errorCode":null,"errorMessage":"unable to reach quorum","messagePattern":"unable to reach quorum","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"dgraph/cmd/zero/tablet.go","lineNumber":171,"sourceCode":"\t}\n\n\t// Feed the outcome of this attempt back to the rebalancer, so it stops re-picking a tablet\n\t// whose moves keep failing.\n\tstart := time.Now()\n\tdefer func() {\n\t\ts.recordMoveResult(predicate, time.Since(start), err)\n\t}()\n\n\ttimeout := moveTimeout(predicateMoveTimeout, tab)\n\tctx, cancel := context.WithTimeout(context.Background(), timeout)\n\tdefer cancel()\n\n\tspan := trace.SpanFromContext(ctx)\n\tdefer span.End()\n\n\t// Ensure that I'm connected to the rest of the Zero group, and am the leader.\n\tif _, err := s.latestMembershipState(ctx); err != nil {\n\t\treturn errors.Wrapf(err, \"unable to reach quorum\")\n\t}\n\tif !s.Node.AmLeader() {\n\t\treturn errors.Errorf(\"I am not the Zero leader\")\n\t}\n\tmsg := fmt.Sprintf(\"Going to move predicate: [%v], size: [ondisk: %v, uncompressed: %v]\"+\n\t\t\" from group %d to %d, timeout: %v\\n\", predicate, humanize.IBytes(uint64(tab.OnDiskBytes)),\n\t\thumanize.IBytes(uint64(tab.UncompressedBytes)), srcGroup, dstGroup, timeout)\n\tglog.Info(msg)\n\tspan.SetAttributes(attribute.String(\"tablet\", predicate))\n\tspan.SetStatus(1, msg)\n\n\t// Block all commits on this predicate. Keep them blocked until we return from this function.\n\tunblock := s.blockTablet(predicate)\n\tdefer unblock()\n\n\t// Get a new timestamp, beyond which we are sure that no new txns would be committed for this\n\t// predicate. Source Alpha leader must reach this timestamp before streaming the data.\n\tids, err := s.Timestamps(ctx, &pb.Num{Val: 1})","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/dgraph/cmd/zero/tablet.go#L153-L189","documentation":"Before streaming data, movePredicate calls s.latestMembershipState(ctx) to force this Zero to sync with the rest of the Zero quorum via a blocking ReadIndex/raft step. If that fails (quorum unreachable, network partition, raft timeouts), the error is wrapped with 'unable to reach quorum'. Zero cannot safely move a predicate without confirming it has an up-to-date view of membership.","triggerScenarios":"Fewer than a majority of Zero nodes reachable (1-node Zero down/restarting, or 3-node Zero with 2 down); network partition between Zero nodes; Zero raft still electing a leader after restart; heavy load causing raft timeouts.","commonSituations":"Operating a multi-node Zero cluster where one replica is down for maintenance; firewall/security-group changes blocking Zero-to-Zero ports (5080 internal, 6080 gRPC); restarting the whole cluster and issuing moves before Zero converges; disk saturation on a Zero node slowing raft.","solutions":["Check all Zero replicas are up: /health and /state on each Zero","Restore network connectivity on Zero internal port 5080 (and 6080) between Zero nodes","Wait for Zero quorum/leader election to complete before retrying moves","Scale Zero back to a majority (restart the down replicas)","Retry the move once the cluster is healthy — the operation is safe to retry"],"exampleFix":"// before\n# issuing move while zero-2 and zero-3 are down\n// after\ncurl -s localhost:6080/health   # verify all zeros healthy first\ndocker start zero-2 zero-3      # restore quorum\ncurl 'localhost:6080/moveTablet?tablet=name&dst_group=2'","handlingStrategy":"retry","validationCode":"# verify all Zero replicas are healthy before moving\nfor z in zero1:6080 zero2:6080 zero3:6080; do\n  curl -sf \"$z/health\" | jq -e '.[0].status == \"healthy\"' || { echo \"$z unhealthy\"; exit 1; }\ndone","typeGuard":null,"tryCatchPattern":"for i in 1 2 3 4 5; do\n  resp=$(curl -s \"localhost:6080/moveTablet?tablet=$TABLET&dst_group=$DST\")\n  [[ \"$(echo \"$resp\" | jq -r .msg)\" == *\"unable to reach quorum\"* ]] && { sleep 15; continue; }\n  break\ndone","preventionTips":["Run an odd number of Zero replicas (>=3) and monitor their health","Keep Zero internal port 5080 open between all Zero nodes","Alert on Zero quorum loss before running rebalancing operations","Wait for full cluster convergence after restarts before issuing moves"],"tags":["dgraph","zero","quorum","raft","network"],"backgroundTag":"quorum-unavailable","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}