{"record":{"id":"946e6abe2d597286","repo":"hashicorp/nomad","slug":"failed-to-unmount-proc-q-w","errorCode":null,"errorMessage":"Failed to unmount proc %q: %w","messagePattern":"Failed to unmount proc %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/allocdir/task_dir_linux.go","lineNumber":32,"sourceCode":"// unmountSpecialDirs unmounts the dev and proc file system from the chroot. No\n// error is returned if the directories do not exist or have already been\n// unmounted.\nfunc (t *TaskDir) unmountSpecialDirs() error {\n\tmErr := new(multierror.Error)\n\tdev := filepath.Join(t.Dir, \"dev\")\n\tif pathExists(dev) {\n\t\tif err := unlinkDir(dev); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to unmount dev %q: %w\", dev, err))\n\t\t} else if err := os.RemoveAll(dev); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to delete dev directory %q: %w\", dev, err))\n\t\t}\n\t}\n\n\t// Unmount proc.\n\tproc := filepath.Join(t.Dir, \"proc\")\n\tif pathExists(proc) {\n\t\tif err := unlinkDir(proc); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to unmount proc %q: %w\", proc, err))\n\t\t} else if err := os.RemoveAll(proc); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to delete proc directory %q: %w\", dev, err))\n\t\t}\n\t}\n\n\treturn mErr.ErrorOrNil()\n}\n","sourceCodeStart":14,"sourceCodeEnd":40,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocdir/task_dir_linux.go#L14-L40","documentation":"When stopping an allocation, unmountSpecialDirs unmounts the task-local proc mount. This error is appended when unlinkDir (which unmounts the proc filesystem on Linux) fails for <allocdir>/proc. The allocation teardown collects it in a multierror but the mount may be left behind.","triggerScenarios":"Unmount (task dir teardown, alloc stop/migration/GC) while <task-dir>/proc exists and unlinkDir's unmount syscall fails — typically EBUSY (something still holds the mount) or EPERM (unprivileged client without mount namespace privileges).","commonSituations":"A task process is still running with its root in the task dir holding /proc busy; Nomad client run without sufficient privileges after a config change (e.g. disabling userns/mount helpers); leftover mounts after a crash; Docker/exec driver leaving zombies.","solutions":["Ensure all task processes in the alloc are terminated (check `ps`/cgroups) so /proc is no longer busy, then retry GC","Run the Nomad agent as root or with CAP_SYS_ADMIN so it can unmount filesystems","Find and release leftover mounts: `grep <alloc-id> /proc/mounts` then `umount` them manually and remove the alloc dir","Reboot or re-provision the client node if mounts are permanently leaked"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before teardown, ensure no live processes hold the task dir\nout, _ := exec.Command(\"grep\", allocID, \"/proc/mounts\").Output()\nif len(out) > 0 {\n  // release mounts / kill lingering processes first\n}","typeGuard":"func isUnmountErr(err error) bool {\n  var le *os.LinkError\n  return errors.As(err, &le)\n}","tryCatchPattern":"if err := ar.Restore(); err != nil {\n  if strings.Contains(err.Error(), \"Failed to unmount proc\") {\n    // mark alloc dir for manual GC; do not retry hot-loop\n  }\n}","preventionTips":["Run the agent with sufficient privileges (root or CAP_SYS_ADMIN) for mount management","Ensure tasks fully terminate before teardown (check cgroups for stragglers)","Periodically audit /proc/mounts for leaked alloc mounts after agent crashes","Avoid sharing the client data dir with processes that mount into alloc dirs"],"tags":["linux","mount","proc","teardown"],"backgroundTag":"mount-busy-unmount-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}