{"record":{"id":"67ba3adc6418495f","repo":"hashicorp/nomad","slug":"failed-to-write-nomad-token-w","errorCode":null,"errorMessage":"failed to write nomad token: %w","messagePattern":"failed to write nomad token: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/identity_hook.go","lineNumber":227,"sourceCode":"// file if requested by the jobsepc.\nfunc (h *identityHook) setDefaultToken() error {\n\ttoken := h.alloc.SignedIdentities[h.task.Name]\n\tif token == \"\" {\n\t\treturn nil\n\t}\n\n\t// Handle internal use and env var\n\th.ts.setNomadToken(token)\n\n\t// Handle file writing\n\tif id := h.task.Identity; id != nil && id.File {\n\t\t// Write token as owner readable only\n\t\ttokenPath := filepath.Join(h.taskDir.SecretsDir, wiTokenFile)\n\t\tif id.Filepath != \"\" {\n\t\t\ttokenPath = filepath.Join(h.taskDir.Dir, id.Filepath)\n\t\t}\n\t\tif err := users.WriteFileFor(tokenPath, []byte(token), h.task.User); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write nomad token: %w\", err)\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// setAltToken takes an alternate workload identity and sets the env var and/or\n// writes the token file as specified by the jobspec.\nfunc (h *identityHook) setAltToken(widspec *structs.WorkloadIdentity, rawJWT string) error {\n\tif widspec.Env {\n\t\th.envBuilder.SetWorkloadToken(widspec.Name, rawJWT)\n\t}\n\n\tif widspec.File {\n\t\ttokenPath := filepath.Join(h.taskDir.SecretsDir, fmt.Sprintf(\"nomad_%s.jwt\", widspec.Name))\n\t\tif widspec.Filepath != \"\" {\n\t\t\ttokenPath = filepath.Join(h.taskDir.Dir, widspec.Filepath)\n\t\t}","sourceCodeStart":209,"sourceCodeEnd":245,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/identity_hook.go#L209-L245","documentation":"setDefaultToken writes the default Nomad workload identity JWT to a file (secrets dir or identity-specified Filepath) using users.WriteFileFor so the task user owns it with restricted permissions. If writing fails, the hook returns 'failed to write nomad token: %w'. The task's Prestart fails, blocking the task from starting, since the token file is required for workloads that read it.","triggerScenarios":"users.WriteFileFor fails — task user does not exist on the client, permission denied on SecretsDir or the custom h.taskDir.Dir/id.Filepath path, disk full, or an id.Filepath pointing into an unwritable/escapes location","commonSituations":"Custom file path configured outside writable dirs, task user changed via `user` stanza to an account not present on the client, read-only or full disk on the client host, or secret dir permissions altered by other hooks/CSI mounts","solutions":["Confirm the task `user` exists on the Nomad client node (or remove the user stanza to use the default)","Check the destination path exists/permissions: SecretsDir must be writable by the nomad agent for the task user","If id.Filepath is set, ensure its directory is created (e.g. via an artifact or template pre-creation) and inside the task dir","Free disk space / fix mount permissions on the client and retry the allocation"],"exampleFix":"// before (directory for custom token path never created)\nidentity {\n  file = true\n  file_path = \"tokens/nomad.jwt\"\n}\n// after (ensure parent dir exists first, e.g. via template)\ntemplate {\n  destination = \"tokens/.keep\"\n  data = \"\"\n}\nidentity {\n  file = true\n  file_path = \"tokens/nomad.jwt\"\n}","handlingStrategy":"try-catch","validationCode":"// preconditions: user exists and target dir writable\nid <task_user> || echo \"user missing on client\"\ntest -w /secrets/dir && echo ok","typeGuard":null,"tryCatchPattern":"// wrap and classify: permission vs disk errors\nif err := users.WriteFileFor(p, tok, user); err != nil {\n    if os.IsPermission(err) { fixPerms() } else if fsErr := statFS(); fsErr != nil { freeDisk() }\n    return fmt.Errorf(\"failed to write nomad token: %w\", err)\n}","preventionTips":["Ensure the task `user` exists on every client the alloc may land on","Create parent dirs for custom identity file paths before Prestart","Monitor client disk usage; keep secrets dir writable","Avoid mounting volumes over the secrets directory"],"tags":["filesystem","workload-identity","permissions"],"backgroundTag":"token-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}