{"record":{"id":"a763978fe90c5824","repo":"hashicorp/nomad","slug":"failed-to-recover-vault-token-from-s-v","errorCode":null,"errorMessage":"failed to recover vault token from %s: %v","messagePattern":"failed to recover vault token from (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/vault_hook.go","lineNumber":183,"sourceCode":"\n\th.vaultConfig = h.vaultConfigsFunc(h.logger)[cluster]\n\tif h.vaultConfig == nil {\n\t\treturn fmt.Errorf(\"No client configuration found for Vault cluster %s\", cluster)\n\t}\n\n\t// Try to recover a token if it was previously written in the secrets\n\t// directory\n\ttoken := \"\"\n\th.privateDirTokenPath = filepath.Join(req.TaskDir.PrivateDir, vaultTokenFile)\n\th.secretsDirTokenPath = filepath.Join(req.TaskDir.SecretsDir, vaultTokenFile)\n\n\t// Handle upgrade path by searching for the previous token in all possible\n\t// paths where the token may be.\n\tfor _, path := range []string{h.privateDirTokenPath, h.secretsDirTokenPath} {\n\t\tdata, err := os.ReadFile(path)\n\t\tif err != nil {\n\t\t\tif !os.IsNotExist(err) {\n\t\t\t\treturn fmt.Errorf(\"failed to recover vault token from %s: %v\", path, err)\n\t\t\t}\n\n\t\t\t// Token file doesn't exist in this path.\n\t\t} else {\n\t\t\t// Store the recovered token\n\t\t\ttoken = string(data)\n\t\t\tbreak\n\t\t}\n\t}\n\n\tduration := 30\n\tif token == \"\" {\n\t\tvar err error\n\t\ttoken, duration, err = h.deriveVaultToken(ctx)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n","sourceCodeStart":165,"sourceCodeEnd":201,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/vault_hook.go#L165-L201","documentation":"As an upgrade/recovery path, the vault hook reads previously persisted Vault tokens from candidate paths (private dir and secrets dir). If os.ReadFile fails with an error other than NotExist (e.g. permission denied, I/O error), Prestart aborts with this wrapped error instead of silently treating the token as absent.","triggerScenarios":"The token file at h.privateDirTokenPath or h.secretsDirTokenPath exists but is unreadable — wrong ownership/permissions from a previous run under a different user, or a disk/filesystem error reading the alloc's secrets directory.","commonSituations":"Alloc directories migrated or restored from backups with wrong uid/gid; running the Nomad agent as a different user than originally; encrypted/readonly filesystem issues on the host data dir.","solutions":["Fix permissions/ownership on the file and its parent directories so the Nomad agent user can read it","Delete the stale token file and restart the task to force fresh token derivation","Check host disk health and mounts for the client data/alloc directory","Run the agent under the same user that originally created the alloc dirs"],"exampleFix":"// before (host shell, as root)\nchown -R root:root /var/lib/nomad\n// after\nchown -R nomad:nomad /var/lib/nomad\nsystemctl restart nomad","handlingStrategy":"try-catch","validationCode":"// check token file readability before Prestart\nfor _, p := range []string{privateDirTokenPath, secretsDirTokenPath} {\n    if f, err := os.Open(p); err != nil && !os.IsNotExist(err) {\n        return fmt.Errorf(\"token file %s unreadable: %w — fix ownership/permissions\", p, err)\n    } else if f != nil {\n        f.Close()\n    }\n}","typeGuard":null,"tryCatchPattern":"data, err := os.ReadFile(path)\nif err != nil && !os.IsNotExist(err) {\n    // distinguish real I/O problems from a simply-absent token\n    return fmt.Errorf(\"failed to recover vault token from %s: %w (check ownership: should be the nomad agent user)\", path, err)\n}","preventionTips":["Never chown/chmod alloc directories to a different user than the agent runs as","Exclude the client data dir from backup-restore flows that change ownership","Keep the nomad agent's system user consistent across upgrades","Monitor disk/mount health on client hosts"],"tags":["vault","filesystem","permissions","tokens","nomad-client"],"backgroundTag":"vault-token-recovery-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}