{"record":{"id":"b029caa25e7fbe5a","repo":"hashicorp/nomad","slug":"failed-to-write-vault-token-v","errorCode":null,"errorMessage":"failed to write vault token: %v","messagePattern":"failed to write vault token: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/vault_hook.go","lineNumber":429,"sourceCode":"\n\treturn token, leaseDuration, nil\n}\n\n// writeToken writes the given token to disk\nfunc (h *vaultHook) writeToken(token string) error {\n\t// Handle upgrade path by first checking if the tasks private directory\n\t// exists. If it doesn't, this allocation probably existed before the\n\t// private directory was introduced, so keep using the secret directory to\n\t// prevent unnecessary errors during task recovery.\n\tif _, err := os.Stat(path.Dir(h.privateDirTokenPath)); os.IsNotExist(err) {\n\t\tif err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write vault token to secrets dir: %v\", err)\n\t\t}\n\t\treturn nil\n\t}\n\n\tif err := os.WriteFile(h.privateDirTokenPath, []byte(token), 0600); err != nil {\n\t\treturn fmt.Errorf(\"failed to write vault token: %v\", err)\n\t}\n\tif !h.vaultBlock.DisableFile {\n\t\tif err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write vault token to secrets dir: %v\", err)\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// withJitter returns when a token should be renewed given its leaseDuration\n// and a randomizer to provide jitter.\n//\n// Leases < 1m will not use jitter.\nfunc withJitter(leaseDuration time.Duration) time.Duration {\n\t// Start trying to renew at half the lease duration to allow ample time\n\t// for latency and retries.\n\trenew := leaseDuration / 2","sourceCodeStart":411,"sourceCodeEnd":447,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/vault_hook.go#L411-L447","documentation":"writeToken writes the token to the task's private directory path (h.privateDirTokenPath, mode 0600) as the primary storage. If os.WriteFile fails there — because the private dir doesn't exist, is unwritable, or the disk has errors — the hook returns this error and the task does not receive a Vault token.","triggerScenarios":"The private directory exists (upgrade-path check passed) but writing the token file to it fails: missing parent directory, wrong ownership/permissions, read-only filesystem, or disk full.","commonSituations":"Task private dir cleaned up externally while alloc runs; Nomad data volume full; container/VM filesystem gone read-only; tmpwatch-like cleaners deleting alloc directories.","solutions":["Ensure the task's private directory exists and is writable by the Nomad client user","Check host disk space and filesystem health (dmesg for read-only remounts)","Restart the allocation to have the taskrunner recreate its directory structure","Audit cron/cleanup jobs that might delete files under the Nomad alloc dir"],"exampleFix":"// host shell\n# check disk and alloc dir\n df -h /var/lib/nomad\n ls -la /var/lib/nomad/alloc/<alloc-id>/<task>/private","handlingStrategy":"validation","validationCode":"// verify private dir exists and is writable before token derivation\nif _, err := os.Stat(privateDir); err != nil {\n    os.MkdirAll(privateDir, 0700)\n}\nprobe := filepath.Join(privateDir, \".probe\")\nif err := os.WriteFile(probe, nil, 0600); err != nil { /* fix fs */ }\nos.Remove(probe)","typeGuard":null,"tryCatchPattern":"if err := writeToken(token); err != nil {\n    if strings.Contains(err.Error(), \"failed to write vault token:\") &&\n       !strings.Contains(err.Error(), \"secrets dir\") {\n        checkDiskAndPerms(privateDirPath)\n        return retryAfterFix()\n    }\n    return err\n}","preventionTips":["Monitor disk space on the Nomad client host (alert at 85%+)","Prevent external tools from deleting the task private directory","Alert on filesystem remounting read-only (dmesg monitoring)","Reschedule allocations whose alloc dir structure was damaged"],"tags":["nomad","vault","filesystem","permissions"],"backgroundTag":"file-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}