hashicorp/nomad · error

self-drain exceeded deadline

Error message

self-drain exceeded deadline

What it means

DrainSelf drains the local node as part of nomad agent leave. It monitors drain progress; when the RPC error is context.DeadlineExceeded or context.Canceled, the self-drain did not finish within its deadline and this error is returned to Leave. The node may be left in a partially drained state.

Source

Thrown at client/drain.go:74

	if drainSpec.Deadline > 0 {
		// if we set this context to the deadline, the server will reach the
		// deadline but not get a chance to record it before this context
		// expires, resulting in spurious errors. So extend the deadline here by
		// a few seconds
		ctx, cancel = context.WithTimeout(context.Background(), drainSpec.Deadline+(5*time.Second))
		defer cancel()
	}
	statusCheckInterval := time.Second

	logger.Info("monitoring self-drain")
	err = c.pollServerForDrainStatus(ctx, statusCheckInterval)
	switch err {
	case nil:
		logger.Debug("self-drain complete")
		return nil
	case context.DeadlineExceeded, context.Canceled:
		logger.Error("self-drain exceeded deadline")
		return fmt.Errorf("self-drain exceeded deadline")
	default:
		logger.Error("could not check node status, falling back to local status checks", "error", err)
	}

	err = c.pollLocalStatusForDrainStatus(ctx, statusCheckInterval, drainSpec)
	if err != nil {
		return fmt.Errorf("self-drain exceeded deadline")
	}

	logger.Debug("self-drain complete")
	return nil
}

// pollServerForDrainStatus will poll the server periodically for the client's
// drain status, returning an error if the context expires or get any error from
// the RPC call. If this function returns nil, the drain was successful.
func (c *Client) pollServerForDrainStatus(ctx context.Context, interval time.Duration) error {
	timer, stop := helper.NewSafeTimer(0)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Increase the drain deadline (nomad node drain -deadline) or the agent leave deadline and retry.
2. Check nomad job status for allocations that failed to migrate — fix scheduling constraints so replacements can be placed.
3. Mark other nodes eligible (nomad node eligibility -enable) so draining allocs have somewhere to go.
4. Retry the drain manually with nomad node drain -enable on the node once blockers are fixed.

Example fix

// before: short deadline on busy node
c.drainingCtx, cancel = context.WithTimeout(ctx, 1*time.Minute)
// after: allow more time
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)

// ensure capacity exists before draining
nodes, _ := client.Nodes().List(nil)
if eligibleHealthy(nodes) < 1 { return errors.New("no eligible nodes to receive drained allocs") }

if err := c.DrainSelf(deadline); err != nil && err.Error() == "self-drain exceeded deadline" {
  // raise deadline, fix stuck allocs, retry drain
  log.Warnf("self-drain deadline hit: %v", err)
}

When it happens

Trigger: The drain deadline (drainSpec.Deadline or the default leave deadline) elapses before all allocations on the node are migrated, or the monitoring context is canceled while allocations remain.

Common situations: Slow/stuck allocations that cannot be rescheduled (no eligible clients for their class, disconnected clients); large node with many allocs and a short leave deadline; scheduler backlog.

Related errors

• invalid drain_on_shutdown config: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8e428a3080f9ee5f. Report an issue: GitHub.