hashicorp/nomad · error

failed to write nomad token: %w

Error message

failed to write nomad token: %w

What it means

setDefaultToken writes the default Nomad workload identity JWT to a file (secrets dir or identity-specified Filepath) using users.WriteFileFor so the task user owns it with restricted permissions. If writing fails, the hook returns 'failed to write nomad token: %w'. The task's Prestart fails, blocking the task from starting, since the token file is required for workloads that read it.

Source

Thrown at client/allocrunner/taskrunner/identity_hook.go:227

// file if requested by the jobsepc.
func (h *identityHook) setDefaultToken() error {
	token := h.alloc.SignedIdentities[h.task.Name]
	if token == "" {
		return nil
	}

	// Handle internal use and env var
	h.ts.setNomadToken(token)

	// Handle file writing
	if id := h.task.Identity; id != nil && id.File {
		// Write token as owner readable only
		tokenPath := filepath.Join(h.taskDir.SecretsDir, wiTokenFile)
		if id.Filepath != "" {
			tokenPath = filepath.Join(h.taskDir.Dir, id.Filepath)
		}
		if err := users.WriteFileFor(tokenPath, []byte(token), h.task.User); err != nil {
			return fmt.Errorf("failed to write nomad token: %w", err)
		}
	}

	return nil
}

// setAltToken takes an alternate workload identity and sets the env var and/or
// writes the token file as specified by the jobspec.
func (h *identityHook) setAltToken(widspec *structs.WorkloadIdentity, rawJWT string) error {
	if widspec.Env {
		h.envBuilder.SetWorkloadToken(widspec.Name, rawJWT)
	}

	if widspec.File {
		tokenPath := filepath.Join(h.taskDir.SecretsDir, fmt.Sprintf("nomad_%s.jwt", widspec.Name))
		if widspec.Filepath != "" {
			tokenPath = filepath.Join(h.taskDir.Dir, widspec.Filepath)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm the task `user` exists on the Nomad client node (or remove the user stanza to use the default)
2. Check the destination path exists/permissions: SecretsDir must be writable by the nomad agent for the task user
3. If id.Filepath is set, ensure its directory is created (e.g. via an artifact or template pre-creation) and inside the task dir
4. Free disk space / fix mount permissions on the client and retry the allocation

Example fix

// before (directory for custom token path never created)
identity {
  file = true
  file_path = "tokens/nomad.jwt"
}
// after (ensure parent dir exists first, e.g. via template)
template {
  destination = "tokens/.keep"
  data = ""
}
identity {
  file = true
  file_path = "tokens/nomad.jwt"
}

// preconditions: user exists and target dir writable
id <task_user> || echo "user missing on client"
test -w /secrets/dir && echo ok

// wrap and classify: permission vs disk errors
if err := users.WriteFileFor(p, tok, user); err != nil {
    if os.IsPermission(err) { fixPerms() } else if fsErr := statFS(); fsErr != nil { freeDisk() }
    return fmt.Errorf("failed to write nomad token: %w", err)
}

When it happens

Trigger: users.WriteFileFor fails — task user does not exist on the client, permission denied on SecretsDir or the custom h.taskDir.Dir/id.Filepath path, disk full, or an id.Filepath pointing into an unwritable/escapes location

Common situations: Custom file path configured outside writable dirs, task user changed via `user` stanza to an account not present on the client, read-only or full disk on the client host, or secret dir permissions altered by other hooks/CSI mounts

Related errors

• failed to write token for identity %q: %w
• plugin not executable
• failed to snapshot %s: %w
• error creating task %q dir: %w
• Couldn't open src file %v: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/67ba3adc6418495f. Report an issue: GitHub.