What causes "Failed to unmount proc %q: %w" in hashicorp/nomad?

Ensure all task processes in the alloc are terminated (check `ps`/cgroups) so /proc is no longer busy, then retry GC

Failed to unmount proc %q: %w

Error message

Failed to unmount proc %q: %w

What it means

When stopping an allocation, unmountSpecialDirs unmounts the task-local proc mount. This error is appended when unlinkDir (which unmounts the proc filesystem on Linux) fails for <allocdir>/proc. The allocation teardown collects it in a multierror but the mount may be left behind.

Source

Thrown at client/allocdir/task_dir_linux.go:32

// unmountSpecialDirs unmounts the dev and proc file system from the chroot. No
// error is returned if the directories do not exist or have already been
// unmounted.
func (t *TaskDir) unmountSpecialDirs() error {
	mErr := new(multierror.Error)
	dev := filepath.Join(t.Dir, "dev")
	if pathExists(dev) {
		if err := unlinkDir(dev); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to unmount dev %q: %w", dev, err))
		} else if err := os.RemoveAll(dev); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to delete dev directory %q: %w", dev, err))
		}
	}

	// Unmount proc.
	proc := filepath.Join(t.Dir, "proc")
	if pathExists(proc) {
		if err := unlinkDir(proc); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to unmount proc %q: %w", proc, err))
		} else if err := os.RemoveAll(proc); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to delete proc directory %q: %w", dev, err))
		}
	}

	return mErr.ErrorOrNil()
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Ensure all task processes in the alloc are terminated (check `ps`/cgroups) so /proc is no longer busy, then retry GC
Run the Nomad agent as root or with CAP_SYS_ADMIN so it can unmount filesystems
Find and release leftover mounts: `grep <alloc-id> /proc/mounts` then `umount` them manually and remove the alloc dir
Reboot or re-provision the client node if mounts are permanently leaked

Defensive patterns

Strategy: validation

Validate before calling

// Before teardown, ensure no live processes hold the task dir
out, _ := exec.Command("grep", allocID, "/proc/mounts").Output()
if len(out) > 0 {
  // release mounts / kill lingering processes first
}

Type guard

func isUnmountErr(err error) bool {
  var le *os.LinkError
  return errors.As(err, &le)
}

Try / catch

if err := ar.Restore(); err != nil {
  if strings.Contains(err.Error(), "Failed to unmount proc") {
    // mark alloc dir for manual GC; do not retry hot-loop
  }
}

Prevention

Run the agent with sufficient privileges (root or CAP_SYS_ADMIN) for mount management
Ensure tasks fully terminate before teardown (check cgroups for stragglers)
Periodically audit /proc/mounts for leaked alloc mounts after agent crashes
Avoid sharing the client data dir with processes that mount into alloc dirs

When it happens

Trigger: Unmount (task dir teardown, alloc stop/migration/GC) while <task-dir>/proc exists and unlinkDir's unmount syscall fails — typically EBUSY (something still holds the mount) or EPERM (unprivileged client without mount namespace privileges).

Common situations: A task process is still running with its root in the task dir holding /proc busy; Nomad client run without sufficient privileges after a config change (e.g. disabling userns/mount helpers); leftover mounts after a crash; Docker/exec driver leaving zombies.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/946e6abe2d597286. Report an issue: GitHub.