hashicorp/nomad · error

failed to write Consul SI token: %w

Error message

failed to write Consul SI token: %w

What it means

Nomad's Consul SI (service identity) task hook wraps the token for the workload into a file (consul_token) inside the task's token directory during Prestart. This error means the os.WriteFile of the Consul SI token SecretID to that file failed, and the underlying OS error is wrapped with %w. It is accumulated into a multi-error rather than returned immediately, so Prestart can report all token write failures at once.

Source

Thrown at client/allocrunner/taskrunner/consul_hook.go:78

		for tokenName, token := range t {
			s := strings.SplitN(tokenName, "/", 2)
			if len(s) < 2 {
				continue
			}
			identity := s[0]
			taskName := s[1]
			// do not write tokens that do not belong to any of this task's
			// identities
			if taskName != h.task.Name || !slices.ContainsFunc(
				h.task.Identities,
				func(id *structs.WorkloadIdentity) bool { return id.Name == identity }) &&
				identity != h.task.Identity.Name {
				continue
			}

			tokenPath := filepath.Join(h.tokenDir, consulTokenFilename)
			if err := os.WriteFile(tokenPath, []byte(token.SecretID), consulTokenFilePerms); err != nil {
				mErr.Errors = append(mErr.Errors, fmt.Errorf("failed to write Consul SI token: %w", err))
			}

			env := map[string]string{
				"CONSUL_TOKEN":      token.SecretID,
				"CONSUL_HTTP_TOKEN": token.SecretID,
			}

			resp.Env = env
		}
	}

	return mErr.ErrorOrNil()
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped underlying error (use errors.Unwrap / %v of the multi-error) to identify the OS cause
2. Verify the task's token directory exists and is writable by the Nomad agent user
3. Check disk space and mount ro flags on the client's data/state volume
4. Inspect SELinux/AppAudit policy denials if using enforcing security modules
5. Retry the allocation after fixing storage; the hook re-runs on restart

Example fix

// before: token dir assumed to exist
os.WriteFile(tokenPath, []byte(token.SecretID), consulTokenFilePerms)
// after: ensure directory exists first
os.MkdirAll(h.tokenDir, 0o700)
os.WriteFile(tokenPath, []byte(token.SecretID), consulTokenFilePerms)

// before Prestart relies on the token dir
if info, err := os.Stat(tokenDir); err != nil || !info.IsDir() {
    return fmt.Errorf("consul token dir %s unavailable: %w", tokenDir, err)
}

if err := hook.Prestart(req); err != nil {
    var pathErr *os.PathError
    if errors.As(err, &pathErr) {
        log.Printf("token write failed on %s: %v", pathErr.Path, pathErr.Err)
    }
}

When it happens

Trigger: os.WriteFile(filepath.Join(h.tokenDir, consulTokenFilename), ...) fails: the token directory does not exist, has wrong permissions, the disk is full or read-only, or the task's token dir was removed mid-prestart. Triggered in Prestart when a workload identity of kind consul matches the task and the hook tries to materialize the token file.

Common situations: Disk full on the client data volume; tokenDir not created due to earlier filesystem errors; running with a chroot/env where the path is not writable; SELinux/AppArmor denying writes to the token directory.

Understand the failure class

Background: "Permission denied" / "Failed to write" file errors: why a library can't write its files to disk (EACCES, EPERM, ENOSPC) and how to fix them — this error's family across 43 libraries.

Related errors

• failed to remove alloc dir %q: %w
• Failed to make the alloc directory %v: %w
• Failed to lookup nobody user: %v
• Failed to create task mount directory: %v
• Failed to mount task dir: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4a07408636e26aa0. Report an issue: GitHub.