{"record":{"id":"83cdfe5f594f1982","repo":"hashicorp/nomad","slug":"drain-for-task-group-q-failed-v","errorCode":null,"errorMessage":"drain for task group %q failed: %v","messagePattern":"drain for task group %q failed: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/drainer/watch_jobs.go","lineNumber":334,"sourceCode":"\t\tif tg.Migrate != nil || batch {\n\t\t\ttaskGroups[tg.Name] = tg\n\t\t}\n\t}\n\n\t// Sort the allocations by TG\n\ttgAllocs := make(map[string][]*structs.Allocation, len(taskGroups))\n\tfor _, alloc := range allocs {\n\t\tif _, ok := taskGroups[alloc.TaskGroup]; !ok {\n\t\t\tcontinue\n\t\t}\n\n\t\ttgAllocs[alloc.TaskGroup] = append(tgAllocs[alloc.TaskGroup], alloc)\n\t}\n\n\tfor name, tg := range taskGroups {\n\t\tallocs := tgAllocs[name]\n\t\tif err := handleTaskGroup(snap, batch, tg, allocs, lastHandledIndex, r); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"drain for task group %q failed: %v\", name, err)\n\t\t}\n\t}\n\n\treturn r, nil\n}\n\n// handleTaskGroup takes the state of a draining task group and computes the\n// desired actions. For batch jobs we only notify when they have been migrated\n// and never mark them for drain. Batch jobs are allowed to complete up until\n// the deadline, after which they are force killed.\nfunc handleTaskGroup(snap *state.StateSnapshot, batch bool, tg *structs.TaskGroup,\n\tallocs []*structs.Allocation, lastHandledIndex uint64, result *jobResult) error {\n\n\t// Determine how many allocations can be drained\n\tdrainingNodes := make(map[string]bool, 4)\n\thealthy := 0\n\tremainingDrainingAlloc := false\n\tvar drainable []*structs.Allocation","sourceCodeStart":316,"sourceCodeEnd":352,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/drainer/watch_jobs.go#L316-L352","documentation":"watch_jobs.handleJob wraps errors from per-task-group drain handling. When handleTaskGroup fails for a specific task group (e.g. an error reading allocations or computing the drain result for that group), Nomad wraps it with the group name so the operator can tell which group's drain failed.","triggerScenarios":"watch -> handleJob iterates task groups and calls handleTaskGroup; any error it returns for group `name` is wrapped as 'drain for task group %q failed: %v'.","commonSituations":"State store read failures while a node is draining; allocations for the group disappearing mid-drain (stopped/cancelled); malformed job/porch data encountered during drain reconciliation; leader transitions during drain handling.","solutions":["Read the wrapped inner error to identify the underlying cause for the named task group","Check the job and allocations with `nomad job status <job>` / `nomad alloc status` for failing allocs","Retry or re-trigger the drain (nomad node drain -enable) to reconcile the group","Inspect server logs for state store errors; update Nomad if the inner error points to a known drain bug"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before triggering drain handling, verify allocations are still present\nallocs, err := state.AllocsByJob(nil, jobNamespace, jobID, false)\n// handleTaskGroup expects non-nil snapshot and matching group allocs","typeGuard":null,"tryCatchPattern":"try {\n    drainer.handleJob(job)\n} catch (e) {\n    const m = /drain for task group \"([^\"]+)\" failed: (.+)/.exec(e.message)\n    if (m) {\n        // m[1] = task group name, m[2] = inner cause; retry once then alert\n        await retry(() => drainer.handleJob(job), {times: 1})\n    } else { throw e }\n}","preventionTips":["Parse the wrapped inner error to find the real cause before retrying","Avoid cancelling jobs/allocs mid-drain so snapshots stay consistent","Monitor server logs for state store errors during node drains"],"tags":["nomad","node-drain","task-group","wrapped-error"],"backgroundTag":"task-group-drain-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}