{"record":{"id":"df00537cae4a8586","repo":"hashicorp/nomad","slug":"failed-to-write-token-for-identity-q-w","errorCode":null,"errorMessage":"failed to write token for identity %q: %w","messagePattern":"failed to write token for identity %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/identity_hook.go","lineNumber":247,"sourceCode":"\t}\n\n\treturn nil\n}\n\n// setAltToken takes an alternate workload identity and sets the env var and/or\n// writes the token file as specified by the jobspec.\nfunc (h *identityHook) setAltToken(widspec *structs.WorkloadIdentity, rawJWT string) error {\n\tif widspec.Env {\n\t\th.envBuilder.SetWorkloadToken(widspec.Name, rawJWT)\n\t}\n\n\tif widspec.File {\n\t\ttokenPath := filepath.Join(h.taskDir.SecretsDir, fmt.Sprintf(\"nomad_%s.jwt\", widspec.Name))\n\t\tif widspec.Filepath != \"\" {\n\t\t\ttokenPath = filepath.Join(h.taskDir.Dir, widspec.Filepath)\n\t\t}\n\t\tif err := users.WriteFileFor(tokenPath, []byte(rawJWT), h.task.User); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to write token for identity %q: %w\", widspec.Name, err)\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// Stop implements interfaces.TaskStopHook\nfunc (h *identityHook) Stop(context.Context, *interfaces.TaskStopRequest, *interfaces.TaskStopResponse) error {\n\th.stop()\n\treturn nil\n}\n\n// Shutdown implements interfaces.ShutdownHook\nfunc (h *identityHook) Shutdown() {\n\th.stop()\n}\n","sourceCodeStart":229,"sourceCodeEnd":264,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/identity_hook.go#L229-L264","documentation":"setAltToken writes a non-default workload identity's JWT to nomad_<name>.jwt in the secrets dir (or widspec.Filepath) when the identity is marked File: true. Failure of users.WriteFileFor yields 'failed to write token for identity %q: %w'. This occurs during identity watch/rotation in watchIdentity, so a failure can abort delivering a rotated token to the running task.","triggerScenarios":"users.WriteFileFor fails for an alternate identity file — task user missing, permission denied on SecretsDir or the custom widspec.Filepath parent directory, disk full, or path unwritable after mount changes","commonSituations":"identity block with file = true and a file_path whose parent directory does not exist, task `user` not provisioned on client, read-only secrets dir after a CSI/volume mount shadowing it, disk exhaustion on the node","solutions":["Ensure the task user exists on the client or drop the `user` stanza","Create the parent directory for the custom file_path (template/artifact pre-creation) and confirm it is inside the task dir","Check SecretsDir is not shadowed by a mounted volume and is writable by the nomad agent and task user","Free disk space / repair filesystem permissions, then restart the allocation"],"exampleFix":"// before\nidentity {\n  name = \"vault\"\n  file = true\n  file_path = \"missing_dir/vault.jwt\"\n}\n// after\ntemplate {\n  destination = \"missing_dir/.keep\"\n  data = \"\"\n}\nidentity {\n  name = \"vault\"\n  file = true\n  file_path = \"missing_dir/vault.jwt\"\n}","handlingStrategy":"try-catch","validationCode":"// preconditions for alternate identity file\nid <task_user> || echo \"user missing\"\n[ -d \"$(dirname tokenPath)\" ] || mkdir -p \"$(dirname tokenPath)\"","typeGuard":null,"tryCatchPattern":"// handle write failure during rotation without losing the old token\nif err := users.WriteFileFor(p, rawJWT, user); err != nil {\n    log.Warn(\"token write failed, retaining previous token\", \"err\", err)\n    return fmt.Errorf(\"failed to write token for identity %q: %w\", name, err)\n}","preventionTips":["Set file_path only inside the task dir with a pre-created parent directory","Provision the task user on all client nodes","Watch for mounts shadowing the secrets dir","Alert on client disk-full conditions"],"tags":["filesystem","workload-identity","permissions"],"backgroundTag":"token-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}