{"record":{"id":"8e428a3080f9ee5f","repo":"hashicorp/nomad","slug":"self-drain-exceeded-deadline","errorCode":null,"errorMessage":"self-drain exceeded deadline","messagePattern":"self-drain exceeded deadline","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/drain.go","lineNumber":74,"sourceCode":"\tif drainSpec.Deadline > 0 {\n\t\t// if we set this context to the deadline, the server will reach the\n\t\t// deadline but not get a chance to record it before this context\n\t\t// expires, resulting in spurious errors. So extend the deadline here by\n\t\t// a few seconds\n\t\tctx, cancel = context.WithTimeout(context.Background(), drainSpec.Deadline+(5*time.Second))\n\t\tdefer cancel()\n\t}\n\tstatusCheckInterval := time.Second\n\n\tlogger.Info(\"monitoring self-drain\")\n\terr = c.pollServerForDrainStatus(ctx, statusCheckInterval)\n\tswitch err {\n\tcase nil:\n\t\tlogger.Debug(\"self-drain complete\")\n\t\treturn nil\n\tcase context.DeadlineExceeded, context.Canceled:\n\t\tlogger.Error(\"self-drain exceeded deadline\")\n\t\treturn fmt.Errorf(\"self-drain exceeded deadline\")\n\tdefault:\n\t\tlogger.Error(\"could not check node status, falling back to local status checks\", \"error\", err)\n\t}\n\n\terr = c.pollLocalStatusForDrainStatus(ctx, statusCheckInterval, drainSpec)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"self-drain exceeded deadline\")\n\t}\n\n\tlogger.Debug(\"self-drain complete\")\n\treturn nil\n}\n\n// pollServerForDrainStatus will poll the server periodically for the client's\n// drain status, returning an error if the context expires or get any error from\n// the RPC call. If this function returns nil, the drain was successful.\nfunc (c *Client) pollServerForDrainStatus(ctx context.Context, interval time.Duration) error {\n\ttimer, stop := helper.NewSafeTimer(0)","sourceCodeStart":56,"sourceCodeEnd":92,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/drain.go#L56-L92","documentation":"DrainSelf drains the local node as part of nomad agent leave. It monitors drain progress; when the RPC error is context.DeadlineExceeded or context.Canceled, the self-drain did not finish within its deadline and this error is returned to Leave. The node may be left in a partially drained state.","triggerScenarios":"The drain deadline (drainSpec.Deadline or the default leave deadline) elapses before all allocations on the node are migrated, or the monitoring context is canceled while allocations remain.","commonSituations":"Slow/stuck allocations that cannot be rescheduled (no eligible clients for their class, disconnected clients); large node with many allocs and a short leave deadline; scheduler backlog.","solutions":["Increase the drain deadline (nomad node drain -deadline) or the agent leave deadline and retry.","Check nomad job status for allocations that failed to migrate — fix scheduling constraints so replacements can be placed.","Mark other nodes eligible (nomad node eligibility -enable) so draining allocs have somewhere to go.","Retry the drain manually with nomad node drain -enable on the node once blockers are fixed."],"exampleFix":"// before: short deadline on busy node\nc.drainingCtx, cancel = context.WithTimeout(ctx, 1*time.Minute)\n// after: allow more time\nctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)\n","handlingStrategy":"retry","validationCode":"// ensure capacity exists before draining\nnodes, _ := client.Nodes().List(nil)\nif eligibleHealthy(nodes) < 1 { return errors.New(\"no eligible nodes to receive drained allocs\") }","typeGuard":null,"tryCatchPattern":"if err := c.DrainSelf(deadline); err != nil && err.Error() == \"self-drain exceeded deadline\" {\n  // raise deadline, fix stuck allocs, retry drain\n  log.Warnf(\"self-drain deadline hit: %v\", err)\n}","preventionTips":["Set a generous drain deadline appropriate for node size","Mark other nodes eligible so migrations can place","Check for constrained jobs that can only run on the draining node"],"tags":["drain","deadline","node-lifecycle"],"backgroundTag":"drain-deadline-exceeded","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}