{"record":{"id":"4d098757311834d2","repo":"hashicorp/nomad","slug":"failed-to-write-vault-token-to-disk-w","errorCode":null,"errorMessage":"failed to write Vault token to disk: %w","messagePattern":"failed to write Vault token to disk: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/vault_hook.go","lineNumber":263,"sourceCode":"\t\t\t\t\t\tstructs.NewTaskEvent(structs.TaskKilling).\n\t\t\t\t\t\t\tSetFailsTask().\n\t\t\t\t\t\t\tSetDisplayMessage(fmt.Sprintf(\"Vault: %v\", err)))\n\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}\n\n// handleRenewalFailure attempts to get a new Vault token and triggers any change_mode\nfunc (h *vaultHook) handleRenewalFailure(ctx context.Context) (string, time.Duration, error) {\n\ttoken, duration, err := h.deriveVaultToken(ctx)\n\tif err != nil {\n\t\treturn \"\", 0, err\n\t}\n\tif err := h.writeToken(token); err != nil {\n\t\treturn \"\", 0, fmt.Errorf(\"failed to write Vault token to disk: %w\", err)\n\t}\n\n\tvar event *structs.TaskEvent\n\tswitch h.vaultBlock.ChangeMode {\n\tcase structs.VaultChangeModeSignal:\n\t\ts, err := signals.Parse(h.vaultBlock.ChangeSignal)\n\t\tif err != nil {\n\t\t\treturn \"\", 0, fmt.Errorf(\"failed to parse signal: %w\", err)\n\t\t}\n\n\t\tevent := structs.NewTaskEvent(structs.TaskSignaling).\n\t\t\tSetTaskSignal(s).SetDisplayMessage(\"Vault: new Vault token acquired\")\n\t\tif err := h.lifecycle.Signal(event, h.vaultBlock.ChangeSignal); err != nil {\n\t\t\treturn \"\", 0, fmt.Errorf(\"failed to send signal: %w\", err)\n\t\t}\n\tcase structs.VaultChangeModeRestart:\n\t\tevent = structs.NewTaskEvent(structs.TaskRestartSignal).\n\t\t\tSetDisplayMessage(\"Vault: new Vault token acquired\")","sourceCodeStart":245,"sourceCodeEnd":281,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/vault_hook.go#L245-L281","documentation":"When a Vault token renewal fails, handleRenewalFailure derives a brand-new token and persists it via writeToken before signaling/restarting per the vault block's change_mode. If writing the new token to the task's secrets directory fails (write error, missing directory, permissions), the recovery aborts with this wrapped error and the task is ultimately killed or evented.","triggerScenarios":"Vault renewal expired/failed and the fallback re-derivation succeeded, but writeToken could not persist the token — e.g. the task's secrets dir was removed, disk full, or permissions were lost mid-task.","commonSituations":"Disk full on the client host; alloc directory garbage-collected while the task ran; permission changes from running under a different user; NFS/readonly mount for the alloc dir.","solutions":["Free disk space on the client host and restart the task","Verify the task's secrets directory exists and is writable by the Nomad agent user","Check vault_hook/token write logs for the inner error wrapped by this message","Ensure consistent agent user/permissions on the alloc directories"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// ensure secrets dir is writable before relying on token persistence\ninfo, err := os.Stat(taskDir.SecretsDir)\nif err != nil || !info.IsDir() {\n    return fmt.Errorf(\"secrets dir missing/not a dir: %w\", err)\n}\nif err := os.WriteFile(filepath.Join(taskDir.SecretsDir, \".probe\"), []byte(\"x\"), 0o600); err != nil {\n    return fmt.Errorf(\"secrets dir not writable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := h.writeToken(token); err != nil {\n    return \"\", 0, fmt.Errorf(\"failed to write Vault token to disk: %w (verify disk space and secrets dir permissions)\", err)\n}","preventionTips":["Monitor client disk usage; full disks break token writes","Don't GC alloc directories for running tasks","Keep agent user permissions consistent across the fleet","Alert on vault renewal-failure events so re-derivation paths are exercised"],"tags":["vault","filesystem","tokens","renewal","nomad-client"],"backgroundTag":"vault-token-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}