What causes "Failed to delete proc directory %q: %w" in hashicorp/nomad?

Check ownership/permissions of the alloc dir tree and chown/chmod so the Nomad client user can delete it

Failed to delete proc directory %q: %w

Error message

Failed to delete proc directory %q: %w

What it means

Companion to the proc unmount step: after unlinkDir succeeds, os.RemoveAll removes the now-empty proc directory inside the task dir. This error is appended when that removal fails, so the directory remains on disk in the allocation dir.

Source

Thrown at client/allocdir/task_dir_linux.go:34

// unmounted.
func (t *TaskDir) unmountSpecialDirs() error {
	mErr := new(multierror.Error)
	dev := filepath.Join(t.Dir, "dev")
	if pathExists(dev) {
		if err := unlinkDir(dev); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to unmount dev %q: %w", dev, err))
		} else if err := os.RemoveAll(dev); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to delete dev directory %q: %w", dev, err))
		}
	}

	// Unmount proc.
	proc := filepath.Join(t.Dir, "proc")
	if pathExists(proc) {
		if err := unlinkDir(proc); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to unmount proc %q: %w", proc, err))
		} else if err := os.RemoveAll(proc); err != nil {
			mErr = multierror.Append(mErr, fmt.Errorf("Failed to delete proc directory %q: %w", dev, err))
		}
	}

	return mErr.ErrorOrNil()
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Check ownership/permissions of the alloc dir tree and chown/chmod so the Nomad client user can delete it
Manually remove the leftover <allocdir>/<task>/proc directory after confirming no mounts remain, then GC the alloc
Verify nothing concurrently mounts into the task dir during shutdown (stop conflicting daemons/backups)
Restart the Nomad client under the same dedicated user that owns the data directory

Example fix

// before
} else if err := os.RemoveAll(proc); err != nil {
    mErr = multierror.Append(mErr, fmt.Errorf("Failed to delete proc directory %q: %w", dev, err))
}
// after
} else if err := os.RemoveAll(proc); err != nil {
    mErr = multierror.Append(mErr, fmt.Errorf("Failed to delete proc directory %q: %w", proc, err))
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Verify the client user can delete the alloc tree before relying on GC
procDir := filepath.Join(allocDir, taskName, "proc")
if _, err := os.Stat(procDir); err == nil {
  if err := os.Chmod(filepath.Dir(procDir), 0700); err != nil {
    return fmt.Errorf("cannot manage task dir: %w", err)
  }
}

Try / catch

if err := ar.Restore(); err != nil {
  if strings.Contains(err.Error(), "Failed to delete proc directory") {
    // queue manual cleanup: umount + rm -rf <allocdir>
  }
}

Prevention

Run the agent under one dedicated user that owns data_dir end to end
Stop backup/monitoring daemons from touching alloc dirs during teardown
Re-check ownership after OS upgrades or user migrations
Manually GC leftover dirs when you see this error repeatedly

When it happens

Trigger: Unmount/teardown path on Linux: unlinkDir(proc) succeeded but os.RemoveAll(proc) failed — typically EACCES/EPERM (parent task dir not writable by the client user) or the directory became non-empty (new mount/process re-created contents between calls).

Common situations: Nomad client data dir permissions changed (run by different user than previously); external monitoring/backup processes touching the alloc dir during teardown; a racing process re-mounting proc into the task dir.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d7eda7d1d960dc56. Report an issue: GitHub.