What causes "drain for task group %q failed: %v" in hashicorp/nomad?

Read the wrapped inner error to identify the underlying cause for the named task group

drain for task group %q failed: %v

Error message

drain for task group %q failed: %v

What it means

watch_jobs.handleJob wraps errors from per-task-group drain handling. When handleTaskGroup fails for a specific task group (e.g. an error reading allocations or computing the drain result for that group), Nomad wraps it with the group name so the operator can tell which group's drain failed.

Source

Thrown at nomad/drainer/watch_jobs.go:334

		if tg.Migrate != nil || batch {
			taskGroups[tg.Name] = tg
		}
	}

	// Sort the allocations by TG
	tgAllocs := make(map[string][]*structs.Allocation, len(taskGroups))
	for _, alloc := range allocs {
		if _, ok := taskGroups[alloc.TaskGroup]; !ok {
			continue
		}

		tgAllocs[alloc.TaskGroup] = append(tgAllocs[alloc.TaskGroup], alloc)
	}

	for name, tg := range taskGroups {
		allocs := tgAllocs[name]
		if err := handleTaskGroup(snap, batch, tg, allocs, lastHandledIndex, r); err != nil {
			return nil, fmt.Errorf("drain for task group %q failed: %v", name, err)
		}
	}

	return r, nil
}

// handleTaskGroup takes the state of a draining task group and computes the
// desired actions. For batch jobs we only notify when they have been migrated
// and never mark them for drain. Batch jobs are allowed to complete up until
// the deadline, after which they are force killed.
func handleTaskGroup(snap *state.StateSnapshot, batch bool, tg *structs.TaskGroup,
	allocs []*structs.Allocation, lastHandledIndex uint64, result *jobResult) error {

	// Determine how many allocations can be drained
	drainingNodes := make(map[string]bool, 4)
	healthy := 0
	remainingDrainingAlloc := false
	var drainable []*structs.Allocation

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the wrapped inner error to identify the underlying cause for the named task group
Check the job and allocations with `nomad job status <job>` / `nomad alloc status` for failing allocs
Retry or re-trigger the drain (nomad node drain -enable) to reconcile the group
Inspect server logs for state store errors; update Nomad if the inner error points to a known drain bug

Defensive patterns

Strategy: retry

Validate before calling

// before triggering drain handling, verify allocations are still present
allocs, err := state.AllocsByJob(nil, jobNamespace, jobID, false)
// handleTaskGroup expects non-nil snapshot and matching group allocs

Try / catch

try {
    drainer.handleJob(job)
} catch (e) {
    const m = /drain for task group "([^"]+)" failed: (.+)/.exec(e.message)
    if (m) {
        // m[1] = task group name, m[2] = inner cause; retry once then alert
        await retry(() => drainer.handleJob(job), {times: 1})
    } else { throw e }
}

Prevention

Parse the wrapped inner error to find the real cause before retrying
Avoid cancelling jobs/allocs mid-drain so snapshots stay consistent
Monitor server logs for state store errors during node drains

When it happens

Trigger: watch -> handleJob iterates task groups and calls handleTaskGroup; any error it returns for group `name` is wrapped as 'drain for task group %q failed: %v'.

Common situations: State store read failures while a node is draining; allocations for the group disappearing mid-drain (stopped/cancelled); malformed job/porch data encountered during drain reconciliation; leader transitions during drain handling.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/83cdfe5f594f1982. Report an issue: GitHub.