{"record":{"id":"a2e5e43698c564be","repo":"hashicorp/nomad","slug":"failed-to-create-task-mount-directory-v","errorCode":null,"errorMessage":"Failed to create task mount directory: %v","messagePattern":"Failed to create task mount directory: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocdir/task_dir.go","lineNumber":192,"sourceCode":"\t\t}\n\t}\n\n\t// Only bind mount the task alloc/task dirs to the client.mounts_dir/<task>\n\tif fsi == fsisolation.Unveil {\n\t\tuid, gid, _, err := dynamic.LookupUser(username)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"Failed to lookup user: %v\", err)\n\t\t}\n\n\t\tnobodyUID, nobodyGID, _, err := dynamic.LookupUser(\"nobody\")\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"Failed to lookup nobody user: %v\", err)\n\t\t}\n\n\t\t// create the task unique directory under the client mounts path\n\t\tparent := filepath.Dir(t.MountsAllocDir)\n\t\tif err = os.MkdirAll(parent, fileMode710); err != nil {\n\t\t\treturn fmt.Errorf(\"Failed to create task mount directory: %v\", err)\n\t\t}\n\t\tif err = os.Chown(parent, uid, gid); err != nil {\n\t\t\treturn fmt.Errorf(\"Failed to chown task mount directory: %v\", err)\n\t\t}\n\n\t\t// create the taskdir mount point\n\t\tif err = mountDir(t.Dir, t.MountsTaskDir, uid, gid, fileMode710); err != nil {\n\t\t\treturn fmt.Errorf(\"Failed to mount task dir: %v\", err)\n\t\t}\n\n\t\t// create the allocdir mount point (owned by nobody)\n\t\tif err = mountDir(filepath.Join(t.AllocDir, \"/alloc\"), t.MountsAllocDir, nobodyUID, nobodyGID, fileMode777); err != nil {\n\t\t\treturn fmt.Errorf(\"Failed to mount alloc dir: %v\", err)\n\t\t}\n\n\t\t// create the secretsdir mount point\n\t\tif err = mountDir(t.SecretsDir, t.MountsSecretsDir, uid, gid, fileMode710); err != nil {\n\t\t\treturn fmt.Errorf(\"Failed to mount secrets dir: %v\", err)","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocdir/task_dir.go#L174-L210","documentation":"During TaskDir.Build in Unveil mode, Nomad creates the per-task directory under the client's mounts path (`parent = filepath.Dir(t.MountsAllocDir)`) with mode 0710 via os.MkdirAll. This error wraps a failure of that MkdirAll. It means the parent mount-point tree for the task could not be created.","triggerScenarios":"os.MkdirAll(parent, 0710) fails while TaskDir.Build runs with fsisolation.Unveil — e.g. the client data/mounts directory is missing, on a read-only filesystem, or a non-directory file already occupies a path component.","commonSituations":"client data_dir on a full or read-only disk; data_dir moved without the mounts subtree existing; permission mismatch where the nomad user cannot write its own data dir; leftover file where a directory is expected after an upgrade/crash.","solutions":["Check disk space and writability of the client data_dir (`df -h`, `touch <data_dir>/probe` as the nomad user)","Inspect the wrapped OS error: ENOSPC/EROFS → free space or remount read-write; ENOTDIR → remove the offending file; EACCES → fix ownership of data_dir","Ensure the nomad client process user owns data_dir: `chown -R nomad:nomad <data_dir>`","Clean stale mounts state (`rm -rf <data_dir>/client/mounts/<alloc>`) after stopping the client"],"exampleFix":"// before: data_dir on read-only volume\nFailed to create task mount directory: mkdir /var/nomad/client/mounts/abc: read-only file system\n// after (host)\n$ sudo mount -o remount,rw /var/nomad\n$ sudo chown -R nomad:nomad /var/nomad","handlingStrategy":"validation","validationCode":"// before client start, verify data_dir writable\nif err := os.MkdirAll(filepath.Join(dataDir, \"client\", \"mounts\"), 0o710); err != nil {\n    log.Fatalf(\"data_dir not writable for mounts tree: %v\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := taskDir.Build(fsi, chroot, username); err != nil {\n    var perr *os.PathError\n    if errors.As(err, &perr) && (perr.Err == syscall.ENOSPC || perr.Err == syscall.EROFS) {\n        // alert: fix data_dir volume, then retry alloc\n    }\n    return err\n}","preventionTips":["Monitor data_dir free space and alert before full","Keep data_dir on local, writable storage — never read-only mounts","Ensure the nomad service user owns data_dir recursively","After changing data_dir in config, recreate the directory tree with correct ownership"],"tags":["filesystem","permissions","nomad-client"],"backgroundTag":"mkdir-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}