{"record":{"id":"26b60ce561463094","repo":"dgraph-io/dgraph","slug":"server-is-being-shut-down","errorCode":null,"errorMessage":"Server is being shut down","messagePattern":"Server is being shut down","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"dgraph/cmd/zero/zero.go","lineNumber":33,"sourceCode":"\t\"sync\"\n\t\"time\"\n\n\t\"github.com/golang/glog\"\n\t\"github.com/pkg/errors\"\n\t\"go.opentelemetry.io/otel\"\n\t\"go.opentelemetry.io/otel/attribute\"\n\t\"google.golang.org/protobuf/proto\"\n\n\t\"github.com/dgraph-io/dgo/v250/protos/api\"\n\t\"github.com/dgraph-io/dgraph/v25/conn\"\n\t\"github.com/dgraph-io/dgraph/v25/protos/pb\"\n\t\"github.com/dgraph-io/dgraph/v25/x\"\n\t\"github.com/dgraph-io/ristretto/v2/z\"\n)\n\nvar (\n\temptyConnectionState pb.ConnectionState\n\terrServerShutDown    = errors.New(\"Server is being shut down\")\n)\n\n// Server implements the zero server.\ntype Server struct {\n\tx.SafeMutex\n\tNode *node\n\torc  *Oracle\n\n\tNumReplicas int\n\tstate       *pb.MembershipState\n\tnextRaftId  uint64\n\n\t// nextUint is the uint64 which we can hand out next. See maxLease for the\n\t// max ID leased via Zero quorum.\n\tnextUint    map[pb.NumLeaseType]uint64\n\treadOnlyTs  uint64\n\tleaseLock   sync.Mutex // protects nextUID, nextTxnTs, nextNsID and corresponding proposals.\n\trateLimiter *x.RateLimiter","sourceCodeStart":15,"sourceCodeEnd":51,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/dgraph/cmd/zero/zero.go#L15-L51","documentation":"errServerShutDown is a sentinel error returned by Oracle and StreamMembership when the Zero server is shutting down. It signals callers that the server can no longer serve requests because its state is being torn down.","triggerScenarios":"Any gRPC call to Oracle or StreamMembership that arrives while Zero's Close() is running and the server's done/shutdown flag is set.","commonSituations":"Client requests racing a Zero restart or upgrade; load balancer still routing to a Zero node that is draining; Alpha reconnecting to a Zero being replaced.","solutions":["Retry the request against another healthy Zero node once it is elected leader.","Ensure a load balancer / service discovery removes Zero nodes during shutdown before they stop serving.","Add retry-with-backoff on the client for this transient sentinel error.","If seen at startup, verify the Zero process is not crash-looping due to config or disk issues."],"exampleFix":"// before\nresp, err := client.Oracle(ctx, req)\n// after\nresp, err := client.Oracle(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"Server is being shut down\") {\n    resp, err = retryWithBackoff(otherZeroClient, ctx, req)\n}","handlingStrategy":"retry","validationCode":"if zeroNode.IsShuttingDown() { skipAndTryNextNode() }","typeGuard":null,"tryCatchPattern":"resp, err := client.Oracle(ctx, req)\nif errors.Is(err, errServerShutDown) || strings.Contains(err.Error(), \"being shut down\") {\n    resp, err = retryOnNextZero(ctx, req)\n}","preventionTips":["Put multiple Zero endpoints in Alpha's zero URL list for automatic failover.","Use a load balancer health check that drains Zero nodes before shutdown.","Implement client retry with backoff for transient gRPC Unavailable-style errors.","Avoid restarting Zero during peak mutation traffic."],"tags":["shutdown","zero","sentinel-error","grpc"],"backgroundTag":"server-shutting-down","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}