hashicorp/nomad · error

failed to write token for identity %q: %w

Error message

failed to write token for identity %q: %w

What it means

setAltToken writes a non-default workload identity's JWT to nomad_<name>.jwt in the secrets dir (or widspec.Filepath) when the identity is marked File: true. Failure of users.WriteFileFor yields 'failed to write token for identity %q: %w'. This occurs during identity watch/rotation in watchIdentity, so a failure can abort delivering a rotated token to the running task.

Source

Thrown at client/allocrunner/taskrunner/identity_hook.go:247

	}

	return nil
}

// setAltToken takes an alternate workload identity and sets the env var and/or
// writes the token file as specified by the jobspec.
func (h *identityHook) setAltToken(widspec *structs.WorkloadIdentity, rawJWT string) error {
	if widspec.Env {
		h.envBuilder.SetWorkloadToken(widspec.Name, rawJWT)
	}

	if widspec.File {
		tokenPath := filepath.Join(h.taskDir.SecretsDir, fmt.Sprintf("nomad_%s.jwt", widspec.Name))
		if widspec.Filepath != "" {
			tokenPath = filepath.Join(h.taskDir.Dir, widspec.Filepath)
		}
		if err := users.WriteFileFor(tokenPath, []byte(rawJWT), h.task.User); err != nil {
			return fmt.Errorf("failed to write token for identity %q: %w", widspec.Name, err)
		}
	}

	return nil
}

// Stop implements interfaces.TaskStopHook
func (h *identityHook) Stop(context.Context, *interfaces.TaskStopRequest, *interfaces.TaskStopResponse) error {
	h.stop()
	return nil
}

// Shutdown implements interfaces.ShutdownHook
func (h *identityHook) Shutdown() {
	h.stop()
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure the task user exists on the client or drop the `user` stanza
2. Create the parent directory for the custom file_path (template/artifact pre-creation) and confirm it is inside the task dir
3. Check SecretsDir is not shadowed by a mounted volume and is writable by the nomad agent and task user
4. Free disk space / repair filesystem permissions, then restart the allocation

Example fix

// before
identity {
  name = "vault"
  file = true
  file_path = "missing_dir/vault.jwt"
}
// after
template {
  destination = "missing_dir/.keep"
  data = ""
}
identity {
  name = "vault"
  file = true
  file_path = "missing_dir/vault.jwt"
}

// preconditions for alternate identity file
id <task_user> || echo "user missing"
[ -d "$(dirname tokenPath)" ] || mkdir -p "$(dirname tokenPath)"

// handle write failure during rotation without losing the old token
if err := users.WriteFileFor(p, rawJWT, user); err != nil {
    log.Warn("token write failed, retaining previous token", "err", err)
    return fmt.Errorf("failed to write token for identity %q: %w", name, err)
}

When it happens

Trigger: users.WriteFileFor fails for an alternate identity file — task user missing, permission denied on SecretsDir or the custom widspec.Filepath parent directory, disk full, or path unwritable after mount changes

Common situations: identity block with file = true and a file_path whose parent directory does not exist, task `user` not provisioned on client, read-only secrets dir after a CSI/volume mount shadowing it, disk exhaustion on the node

Related errors

• failed to write nomad token: %w
• plugin not executable
• failed to snapshot %s: %w
• error creating task %q dir: %w
• Couldn't open src file %v: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/df00537cae4a8586. Report an issue: GitHub.