What causes "Exec: %w" in hashicorp/nomad?

Confirm the task/alloc is still running — a dead task's cgroup is gone; re-run exec on a live alloc

Exec: %w

Error message

Exec: %w

What it means

UniversalExecutor.Exec runs an additional command inside the existing task's containment. Before ExecScript it moves the sub-command into the task's stats cgroup via setSubCmdCgroup; any error there is wrapped as 'Exec: %w'. The actual in-container command execution happens afterwards and is not part of this error.

Source

Thrown at drivers/shared/executor/executor.go:448

	}

	// Run the runningFunc hook after the process starts
	if err := running(); err != nil {
		return nil, err
	}

	// Wait on the task process
	go e.wait()
	return &ProcessState{Pid: e.childCmd.Process.Pid, ExitCode: -1, Time: time.Now()}, nil
}

// Exec a command inside a container for exec and java drivers.
func (e *UniversalExecutor) Exec(deadline time.Time, name string, args []string) ([]byte, int, error) {
	ctx, cancel := context.WithDeadline(context.Background(), deadline)
	defer cancel()

	if cleanup, err := e.setSubCmdCgroup(&e.childCmd, e.command.StatsCgroup()); err != nil {
		return nil, 0, fmt.Errorf("Exec: %w", err)
	} else {
		defer cleanup()
	}

	return ExecScript(ctx, e.childCmd.Dir, e.command.Env, e.childCmd.SysProcAttr, e.command.NetworkIsolation, name, args)
}

// ExecScript executes cmd with args and returns the output, exit code, and
// error. Output is truncated to drivers/shared/structs.CheckBufSize
func ExecScript(ctx context.Context, dir string, env []string, attrs *syscall.SysProcAttr,
	netSpec *drivers.NetworkIsolationSpec, name string, args []string) ([]byte, int, error) {

	cmd := exec.CommandContext(ctx, name, args...)

	// Copy runtime environment from the main command
	cmd.SysProcAttr = attrs

	cmd.Dir = dir

View on GitHub (pinned to 482b49bf1a)

Solutions

Confirm the task/alloc is still running — a dead task's cgroup is gone; re-run exec on a live alloc
Check the wrapped %w cause: ENOENT/EACCES on cgroup.procs indicates cleanup or permissions
Ensure /sys/fs/cgroup is writable and the client has privileges (root or delegated cgroups)
Upgrade Nomad if on cgroup v2 with an older release lacking v2 support

Example fix

// before
// nomad alloc exec -job web date   # alloc already dead
// after
// nomad status web && nomad alloc exec -job web date  # target a running alloc

Defensive patterns

Strategy: validation

Validate before calling

func execTargetAlive(allocStatus string, statsCgroup string) error {
  if allocStatus != "running" { return fmt.Errorf("alloc not running: %s", allocStatus) }
  if statsCgroup == "" { return errors.New("no stats cgroup for task") }
  if _, err := os.Stat(filepath.Join(statsCgroup, "cgroup.procs")); err != nil {
    return fmt.Errorf("task cgroup gone (task exited?): %w", err)
  }
  return nil
}

Try / catch

out, code, err := executor.Exec(deadline, "sh", args)
if err != nil {
  if strings.Contains(err.Error(), "Exec:") && strings.Contains(err.Error(), "no such file") {
    return fmt.Errorf("task cgroup disappeared; alloc likely dead: %w", err)
  }
  return err
}

Prevention

Check alloc status before running alloc exec
Avoid exec-ing into tasks right after stop/kill signals
Grant cgroup delegation if the client is non-root
Upgrade for cgroup v2 compatibility on modern hosts

When it happens

Trigger: Exec(deadline, name, args) is called (e.g. `nomad alloc exec`) and setSubCmdCgroup fails writing the child PID into the task's cgroup.procs — cgroup path gone (task exited), cgroup not writable, or StatsCgroup() empty/invalid.

Common situations: Running alloc exec after the task process already exited (its cgroup was cleaned up), exec into tasks on hosts with read-only cgroup mounts, cgroup v2 hosts with older Nomad builds, driver restarted between task start and exec.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/40652cbcfeee7ebd. Report an issue: GitHub.