{"record":{"id":"d7eda7d1d960dc56","repo":"hashicorp/nomad","slug":"failed-to-delete-proc-directory-q-w","errorCode":null,"errorMessage":"Failed to delete proc directory %q: %w","messagePattern":"Failed to delete proc directory %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"client/allocdir/task_dir_linux.go","lineNumber":34,"sourceCode":"// unmounted.\nfunc (t *TaskDir) unmountSpecialDirs() error {\n\tmErr := new(multierror.Error)\n\tdev := filepath.Join(t.Dir, \"dev\")\n\tif pathExists(dev) {\n\t\tif err := unlinkDir(dev); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to unmount dev %q: %w\", dev, err))\n\t\t} else if err := os.RemoveAll(dev); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to delete dev directory %q: %w\", dev, err))\n\t\t}\n\t}\n\n\t// Unmount proc.\n\tproc := filepath.Join(t.Dir, \"proc\")\n\tif pathExists(proc) {\n\t\tif err := unlinkDir(proc); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to unmount proc %q: %w\", proc, err))\n\t\t} else if err := os.RemoveAll(proc); err != nil {\n\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\"Failed to delete proc directory %q: %w\", dev, err))\n\t\t}\n\t}\n\n\treturn mErr.ErrorOrNil()\n}\n","sourceCodeStart":16,"sourceCodeEnd":40,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocdir/task_dir_linux.go#L16-L40","documentation":"Companion to the proc unmount step: after unlinkDir succeeds, os.RemoveAll removes the now-empty proc directory inside the task dir. This error is appended when that removal fails, so the directory remains on disk in the allocation dir.","triggerScenarios":"Unmount/teardown path on Linux: unlinkDir(proc) succeeded but os.RemoveAll(proc) failed — typically EACCES/EPERM (parent task dir not writable by the client user) or the directory became non-empty (new mount/process re-created contents between calls).","commonSituations":"Nomad client data dir permissions changed (run by different user than previously); external monitoring/backup processes touching the alloc dir during teardown; a racing process re-mounting proc into the task dir.","solutions":["Check ownership/permissions of the alloc dir tree and chown/chmod so the Nomad client user can delete it","Manually remove the leftover <allocdir>/<task>/proc directory after confirming no mounts remain, then GC the alloc","Verify nothing concurrently mounts into the task dir during shutdown (stop conflicting daemons/backups)","Restart the Nomad client under the same dedicated user that owns the data directory"],"exampleFix":"// before\n} else if err := os.RemoveAll(proc); err != nil {\n    mErr = multierror.Append(mErr, fmt.Errorf(\"Failed to delete proc directory %q: %w\", dev, err))\n}\n// after\n} else if err := os.RemoveAll(proc); err != nil {\n    mErr = multierror.Append(mErr, fmt.Errorf(\"Failed to delete proc directory %q: %w\", proc, err))\n}","handlingStrategy":"try-catch","validationCode":"// Verify the client user can delete the alloc tree before relying on GC\nprocDir := filepath.Join(allocDir, taskName, \"proc\")\nif _, err := os.Stat(procDir); err == nil {\n  if err := os.Chmod(filepath.Dir(procDir), 0700); err != nil {\n    return fmt.Errorf(\"cannot manage task dir: %w\", err)\n  }\n}","typeGuard":null,"tryCatchPattern":"if err := ar.Restore(); err != nil {\n  if strings.Contains(err.Error(), \"Failed to delete proc directory\") {\n    // queue manual cleanup: umount + rm -rf <allocdir>\n  }\n}","preventionTips":["Run the agent under one dedicated user that owns data_dir end to end","Stop backup/monitoring daemons from touching alloc dirs during teardown","Re-check ownership after OS upgrades or user migrations","Manually GC leftover dirs when you see this error repeatedly"],"tags":["linux","filesystem","teardown","permissions"],"backgroundTag":"directory-removal-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}