dgraph-io/dgraph · warning
the server is in draining mode and client requests will only
Error message
the server is in draining mode and client requests will only be allowed after exiting the mode by sending a GraphQL draining(enable: false) mutation to /admin
What it means
errDrainingMode is a sentinel error returned by HealthCheck while the alpha is in draining mode, entered via the GraphQL draining(enable: true) mutation on /admin. In this mode the server is being gracefully drained (e.g. for a decommission or move) and rejects normal client traffic until draining is explicitly disabled.
Source
Thrown at x/health.go:24
package x
import (
"sync/atomic"
"github.com/golang/glog"
"github.com/pkg/errors"
)
var (
// the drainingMode variable should be accessed through the atomic.Store and atomic.Load
// functions. The value 0 means the draining-mode is disabled, and the value 1 means the
// mode is enabled
drainingMode uint32
extSnapshotStreamingState uint32
healthCheck uint32
errHealth = errors.New("Please retry again, server is not ready to accept requests")
errDrainingMode = errors.New("the server is in draining mode " +
"and client requests will only be allowed after exiting the mode " +
" by sending a GraphQL draining(enable: false) mutation to /admin")
)
// UpdateHealthStatus updates the server's health status so it can start accepting requests.
func UpdateHealthStatus(ok bool) {
setStatus(&healthCheck, ok)
}
// UpdateDrainingMode updates the server's draining mode
func UpdateDrainingMode(enable bool) {
setStatus(&drainingMode, enable)
}
// ExtSnapshotStreamingState updates the server's import mode
func ExtSnapshotStreamingState(enable bool) {
glog.Info("[import] Updating import mode to ", enable)
setStatus(&extSnapshotStreamingState, enable)View on GitHub (pinned to 759e242be6)
Solutions
- Exit draining mode via GraphQL on the /admin endpoint: mutation { draining(enable: false) { response { message } } }
- Verify state by calling HealthCheck or the health endpoint after disabling draining
- Check which operator/automation enabled draining and ensure it disables it when done
- If the node was meant to be removed, route traffic to other alphas instead
Example fix
// before (client retries forever against a drained node)
resp, err := doQuery(ctx, alphaInDrain)
// after
mutation := `{ draining(enable: false) { response { message } } }`
_, err := sendAdminGQL(ctx, "http://alpha:8080/admin", mutation) // then retry the query
resp, err = doQuery(ctx, alphaInDrain) Defensive patterns
Strategy: fallback
Validate before calling
// Check node state before routing traffic
if err := x.HealthCheck(); err != nil && strings.Contains(err.Error(), "draining mode") {
// skip this alpha, route to another node or disable draining first
} Try / catch
err := doRequest(client)
if err != nil && strings.Contains(err.Error(), "draining mode") {
// route to a different alpha, or POST draining(enable:false) to /admin and retry
} Prevention
- Automation that enables draining must always disable it (use defer/finally semantics in scripts)
- Alert when an alpha stays in draining mode beyond an expected maintenance window
- Route traffic away from drained nodes via load-balancer health checks
- Document runbooks so drained nodes are either re-enabled or removed
When it happens
Trigger: Calling HealthCheck while the drainingMode flag is set; sending client traffic to an alpha that an operator put into draining mode with draining(enable: true); requests landing on an alpha mid-decommission/move-tablet flow.
Common situations: Cluster maintenance: operator drained an alpha to move a predicate or remove it and forgot to re-enable; automation that enables draining but does not disable it on failure; stale dashboards/scripts leaving nodes drained.
Related errors
AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01).
Data as JSON: /api/errors/7a955d70fda3920b.
Report an issue: GitHub.