What causes "Server is being shut down" in dgraph-io/dgraph?

Retry the request against another healthy Zero node once it is elected leader.

Server is being shut down

Error message

Server is being shut down

What it means

errServerShutDown is a sentinel error returned by Oracle and StreamMembership when the Zero server is shutting down. It signals callers that the server can no longer serve requests because its state is being torn down.

Source

Thrown at dgraph/cmd/zero/zero.go:33

	"sync"
	"time"

	"github.com/golang/glog"
	"github.com/pkg/errors"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"google.golang.org/protobuf/proto"

	"github.com/dgraph-io/dgo/v250/protos/api"
	"github.com/dgraph-io/dgraph/v25/conn"
	"github.com/dgraph-io/dgraph/v25/protos/pb"
	"github.com/dgraph-io/dgraph/v25/x"
	"github.com/dgraph-io/ristretto/v2/z"
)

var (
	emptyConnectionState pb.ConnectionState
	errServerShutDown    = errors.New("Server is being shut down")
)

// Server implements the zero server.
type Server struct {
	x.SafeMutex
	Node *node
	orc  *Oracle

	NumReplicas int
	state       *pb.MembershipState
	nextRaftId  uint64

	// nextUint is the uint64 which we can hand out next. See maxLease for the
	// max ID leased via Zero quorum.
	nextUint    map[pb.NumLeaseType]uint64
	readOnlyTs  uint64
	leaseLock   sync.Mutex // protects nextUID, nextTxnTs, nextNsID and corresponding proposals.
	rateLimiter *x.RateLimiter

View on GitHub (pinned to 759e242be6)

Solutions

Retry the request against another healthy Zero node once it is elected leader.
Ensure a load balancer / service discovery removes Zero nodes during shutdown before they stop serving.
Add retry-with-backoff on the client for this transient sentinel error.
If seen at startup, verify the Zero process is not crash-looping due to config or disk issues.

Example fix

// before
resp, err := client.Oracle(ctx, req)
// after
resp, err := client.Oracle(ctx, req)
if err != nil && strings.Contains(err.Error(), "Server is being shut down") {
    resp, err = retryWithBackoff(otherZeroClient, ctx, req)
}

Defensive patterns

Strategy: retry

Validate before calling

if zeroNode.IsShuttingDown() { skipAndTryNextNode() }

Try / catch

resp, err := client.Oracle(ctx, req)
if errors.Is(err, errServerShutDown) || strings.Contains(err.Error(), "being shut down") {
    resp, err = retryOnNextZero(ctx, req)
}

Prevention

Put multiple Zero endpoints in Alpha's zero URL list for automatic failover.
Use a load balancer health check that drains Zero nodes before shutdown.
Implement client retry with backoff for transient gRPC Unavailable-style errors.
Avoid restarting Zero during peak mutation traffic.

When it happens

Trigger: Any gRPC call to Oracle or StreamMembership that arrives while Zero's Close() is running and the server's done/shutdown flag is set.

Common situations: Client requests racing a Zero restart or upgrade; load balancer still routing to a Zero node that is draining; Alpha reconnecting to a Zero being replaced.

Related errors

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/26b60ce561463094. Report an issue: GitHub.