{"record":{"id":"fbb129fc55842e7e","repo":"hashicorp/nomad","slug":"failed-to-write-vault-token-to-secrets-dir-v","errorCode":null,"errorMessage":"failed to write vault token to secrets dir: %v","messagePattern":"failed to write vault token to secrets dir: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/vault_hook.go","lineNumber":423,"sourceCode":"\n\t// If the token cannot be renewed, it doesn't matter if the user set\n\t// allow_token_expiration or not, so override the requested behavior\n\tif !renewable {\n\t\th.allowTokenExpiration = true\n\t}\n\n\treturn token, leaseDuration, nil\n}\n\n// writeToken writes the given token to disk\nfunc (h *vaultHook) writeToken(token string) error {\n\t// Handle upgrade path by first checking if the tasks private directory\n\t// exists. If it doesn't, this allocation probably existed before the\n\t// private directory was introduced, so keep using the secret directory to\n\t// prevent unnecessary errors during task recovery.\n\tif _, err := os.Stat(path.Dir(h.privateDirTokenPath)); os.IsNotExist(err) {\n\t\tif err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write vault token to secrets dir: %v\", err)\n\t\t}\n\t\treturn nil\n\t}\n\n\tif err := os.WriteFile(h.privateDirTokenPath, []byte(token), 0600); err != nil {\n\t\treturn fmt.Errorf(\"failed to write vault token: %v\", err)\n\t}\n\tif !h.vaultBlock.DisableFile {\n\t\tif err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write vault token to secrets dir: %v\", err)\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// withJitter returns when a token should be renewed given its leaseDuration\n// and a randomizer to provide jitter.","sourceCodeStart":405,"sourceCodeEnd":441,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/vault_hook.go#L405-L441","documentation":"writeToken persists the Vault token file for the task. On the upgrade path — when the task's private directory does not exist (pre-private-dir allocations) — it falls back to writing the token into the legacy secrets directory with os.WriteFile. If that write fails (permissions, disk full, path missing), this error is returned.","triggerScenarios":"Allocation created before the private token directory was introduced; os.Stat on the private dir returns IsNotExist, then os.WriteFile to h.secretsDirTokenPath fails due to filesystem permissions, a missing secrets dir, or I/O errors.","commonSituations":"Upgraded Nomad clients running old allocations; read-only or full data volumes; secrets dir permissions changed by external tooling (security hardening, SELinux).","solutions":["Check permissions/ownership of the alloc's secrets directory on the client host","Free disk space or resolve I/O errors on the Nomad data volume","Restart the allocation so the taskrunner recreates the private directory and normal write path","Ensure security tooling (SELinux/AppArmor) is not blocking writes to the Nomad alloc dir"],"exampleFix":"// host shell: fix perms on the alloc secrets dir\nchmod u+rwX /var/lib/nomad/alloc/<alloc-id>/<task>/secrets","handlingStrategy":"validation","validationCode":"// before starting allocations, check the secrets dir is writable\nst, err := os.Stat(secretsDir)\nif err != nil || !st.IsDir() { /* fix dir */ }\nf, err := os.OpenFile(filepath.Join(secretsDir, \".probe\"), os.O_CREATE|os.O_WRONLY, 0666)\nif err != nil { /* permissions problem */ }\nf.Close(); os.Remove(filepath.Join(secretsDir, \".probe\"))","typeGuard":null,"tryCatchPattern":"if err := writeToken(token); err != nil {\n    if strings.Contains(err.Error(), \"failed to write vault token\") {\n        // inspect alloc dir perms/disk, then reschedule the alloc\n        return rescheduleAlloc(allocID)\n    }\n    return err\n}","preventionTips":["Keep the Nomad data volume healthy and with free space","Exclude the Nomad alloc dir from cleanup daemons (tmpwatch, bleachbit)","Don't harden/SELinux-restrict the alloc dir paths without testing","Upgrade allocations to recreate the private directory when feasible"],"tags":["nomad","vault","filesystem","permissions"],"backgroundTag":"file-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}