{"record":{"id":"7a955d70fda3920b","repo":"dgraph-io/dgraph","slug":"the-server-is-in-draining-mode-and-client-requests","errorCode":null,"errorMessage":"the server is in draining mode and client requests will only be allowed after exiting the mode  by sending a GraphQL draining(enable: false) mutation to /admin","messagePattern":"the server is in draining mode and client requests will only be allowed after exiting the mode  by sending a GraphQL draining\\(enable: false\\) mutation to /admin","errorType":"http","errorClass":null,"httpStatus":503,"severity":"warning","filePath":"x/health.go","lineNumber":24,"sourceCode":"package x\n\nimport (\n\t\"sync/atomic\"\n\n\t\"github.com/golang/glog\"\n\t\"github.com/pkg/errors\"\n)\n\nvar (\n\t// the drainingMode variable should be accessed through the atomic.Store and atomic.Load\n\t// functions. The value 0 means the draining-mode is disabled, and the value 1 means the\n\t// mode is enabled\n\tdrainingMode              uint32\n\textSnapshotStreamingState uint32\n\n\thealthCheck     uint32\n\terrHealth       = errors.New(\"Please retry again, server is not ready to accept requests\")\n\terrDrainingMode = errors.New(\"the server is in draining mode \" +\n\t\t\"and client requests will only be allowed after exiting the mode \" +\n\t\t\" by sending a GraphQL draining(enable: false) mutation to /admin\")\n)\n\n// UpdateHealthStatus updates the server's health status so it can start accepting requests.\nfunc UpdateHealthStatus(ok bool) {\n\tsetStatus(&healthCheck, ok)\n}\n\n// UpdateDrainingMode updates the server's draining mode\nfunc UpdateDrainingMode(enable bool) {\n\tsetStatus(&drainingMode, enable)\n}\n\n// ExtSnapshotStreamingState updates the server's import mode\nfunc ExtSnapshotStreamingState(enable bool) {\n\tglog.Info(\"[import] Updating import mode to \", enable)\n\tsetStatus(&extSnapshotStreamingState, enable)","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/x/health.go#L6-L42","documentation":"errDrainingMode is a sentinel error returned by HealthCheck while the alpha is in draining mode, entered via the GraphQL draining(enable: true) mutation on /admin. In this mode the server is being gracefully drained (e.g. for a decommission or move) and rejects normal client traffic until draining is explicitly disabled.","triggerScenarios":"Calling HealthCheck while the drainingMode flag is set; sending client traffic to an alpha that an operator put into draining mode with draining(enable: true); requests landing on an alpha mid-decommission/move-tablet flow.","commonSituations":"Cluster maintenance: operator drained an alpha to move a predicate or remove it and forgot to re-enable; automation that enables draining but does not disable it on failure; stale dashboards/scripts leaving nodes drained.","solutions":["Exit draining mode via GraphQL on the /admin endpoint: mutation { draining(enable: false) { response { message } } }","Verify state by calling HealthCheck or the health endpoint after disabling draining","Check which operator/automation enabled draining and ensure it disables it when done","If the node was meant to be removed, route traffic to other alphas instead"],"exampleFix":"// before (client retries forever against a drained node)\nresp, err := doQuery(ctx, alphaInDrain)\n// after\nmutation := `{ draining(enable: false) { response { message } } }`\n_, err := sendAdminGQL(ctx, \"http://alpha:8080/admin\", mutation) // then retry the query\nresp, err = doQuery(ctx, alphaInDrain)","handlingStrategy":"fallback","validationCode":"// Check node state before routing traffic\nif err := x.HealthCheck(); err != nil && strings.Contains(err.Error(), \"draining mode\") {\n    // skip this alpha, route to another node or disable draining first\n}","typeGuard":null,"tryCatchPattern":"err := doRequest(client)\nif err != nil && strings.Contains(err.Error(), \"draining mode\") {\n    // route to a different alpha, or POST draining(enable:false) to /admin and retry\n}","preventionTips":["Automation that enables draining must always disable it (use defer/finally semantics in scripts)","Alert when an alpha stays in draining mode beyond an expected maintenance window","Route traffic away from drained nodes via load-balancer health checks","Document runbooks so drained nodes are either re-enabled or removed"],"tags":["draining-mode","admin-mutation","maintenance","health-check"],"backgroundTag":"server-draining-mode","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}