dgraph-io/dgraph · warning

the server is in draining mode and client requests will only

Error message

the server is in draining mode and client requests will only be allowed after exiting the mode  by sending a GraphQL draining(enable: false) mutation to /admin

What it means

errDrainingMode is a sentinel error returned by HealthCheck while the alpha is in draining mode, entered via the GraphQL draining(enable: true) mutation on /admin. In this mode the server is being gracefully drained (e.g. for a decommission or move) and rejects normal client traffic until draining is explicitly disabled.

Source

Thrown at x/health.go:24

package x

import (
	"sync/atomic"

	"github.com/golang/glog"
	"github.com/pkg/errors"
)

var (
	// the drainingMode variable should be accessed through the atomic.Store and atomic.Load
	// functions. The value 0 means the draining-mode is disabled, and the value 1 means the
	// mode is enabled
	drainingMode              uint32
	extSnapshotStreamingState uint32

	healthCheck     uint32
	errHealth       = errors.New("Please retry again, server is not ready to accept requests")
	errDrainingMode = errors.New("the server is in draining mode " +
		"and client requests will only be allowed after exiting the mode " +
		" by sending a GraphQL draining(enable: false) mutation to /admin")
)

// UpdateHealthStatus updates the server's health status so it can start accepting requests.
func UpdateHealthStatus(ok bool) {
	setStatus(&healthCheck, ok)
}

// UpdateDrainingMode updates the server's draining mode
func UpdateDrainingMode(enable bool) {
	setStatus(&drainingMode, enable)
}

// ExtSnapshotStreamingState updates the server's import mode
func ExtSnapshotStreamingState(enable bool) {
	glog.Info("[import] Updating import mode to ", enable)
	setStatus(&extSnapshotStreamingState, enable)

View on GitHub (pinned to 759e242be6)

Solutions

  1. Exit draining mode via GraphQL on the /admin endpoint: mutation { draining(enable: false) { response { message } } }
  2. Verify state by calling HealthCheck or the health endpoint after disabling draining
  3. Check which operator/automation enabled draining and ensure it disables it when done
  4. If the node was meant to be removed, route traffic to other alphas instead

Example fix

// before (client retries forever against a drained node)
resp, err := doQuery(ctx, alphaInDrain)
// after
mutation := `{ draining(enable: false) { response { message } } }`
_, err := sendAdminGQL(ctx, "http://alpha:8080/admin", mutation) // then retry the query
resp, err = doQuery(ctx, alphaInDrain)
Defensive patterns

Strategy: fallback

Validate before calling

// Check node state before routing traffic
if err := x.HealthCheck(); err != nil && strings.Contains(err.Error(), "draining mode") {
    // skip this alpha, route to another node or disable draining first
}

Try / catch

err := doRequest(client)
if err != nil && strings.Contains(err.Error(), "draining mode") {
    // route to a different alpha, or POST draining(enable:false) to /admin and retry
}

Prevention

When it happens

Trigger: Calling HealthCheck while the drainingMode flag is set; sending client traffic to an alpha that an operator put into draining mode with draining(enable: true); requests landing on an alpha mid-decommission/move-tablet flow.

Common situations: Cluster maintenance: operator drained an alpha to move a predicate or remove it and forgot to re-enable; automation that enables draining but does not disable it on failure; stale dashboards/scripts leaving nodes drained.

Related errors


AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/7a955d70fda3920b. Report an issue: GitHub.