hashicorp/nomad · error

failed to recover vault token from %s: %v

Error message

failed to recover vault token from %s: %v

What it means

As an upgrade/recovery path, the vault hook reads previously persisted Vault tokens from candidate paths (private dir and secrets dir). If os.ReadFile fails with an error other than NotExist (e.g. permission denied, I/O error), Prestart aborts with this wrapped error instead of silently treating the token as absent.

Source

Thrown at client/allocrunner/taskrunner/vault_hook.go:183

	h.vaultConfig = h.vaultConfigsFunc(h.logger)[cluster]
	if h.vaultConfig == nil {
		return fmt.Errorf("No client configuration found for Vault cluster %s", cluster)
	}

	// Try to recover a token if it was previously written in the secrets
	// directory
	token := ""
	h.privateDirTokenPath = filepath.Join(req.TaskDir.PrivateDir, vaultTokenFile)
	h.secretsDirTokenPath = filepath.Join(req.TaskDir.SecretsDir, vaultTokenFile)

	// Handle upgrade path by searching for the previous token in all possible
	// paths where the token may be.
	for _, path := range []string{h.privateDirTokenPath, h.secretsDirTokenPath} {
		data, err := os.ReadFile(path)
		if err != nil {
			if !os.IsNotExist(err) {
				return fmt.Errorf("failed to recover vault token from %s: %v", path, err)
			}

			// Token file doesn't exist in this path.
		} else {
			// Store the recovered token
			token = string(data)
			break
		}
	}

	duration := 30
	if token == "" {
		var err error
		token, duration, err = h.deriveVaultToken(ctx)
		if err != nil {
			return err
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Fix permissions/ownership on the file and its parent directories so the Nomad agent user can read it
2. Delete the stale token file and restart the task to force fresh token derivation
3. Check host disk health and mounts for the client data/alloc directory
4. Run the agent under the same user that originally created the alloc dirs

Example fix

// before (host shell, as root)
chown -R root:root /var/lib/nomad
// after
chown -R nomad:nomad /var/lib/nomad
systemctl restart nomad

// check token file readability before Prestart
for _, p := range []string{privateDirTokenPath, secretsDirTokenPath} {
    if f, err := os.Open(p); err != nil && !os.IsNotExist(err) {
        return fmt.Errorf("token file %s unreadable: %w — fix ownership/permissions", p, err)
    } else if f != nil {
        f.Close()
    }
}

data, err := os.ReadFile(path)
if err != nil && !os.IsNotExist(err) {
    // distinguish real I/O problems from a simply-absent token
    return fmt.Errorf("failed to recover vault token from %s: %w (check ownership: should be the nomad agent user)", path, err)
}

When it happens

Trigger: The token file at h.privateDirTokenPath or h.secretsDirTokenPath exists but is unreadable — wrong ownership/permissions from a previous run under a different user, or a disk/filesystem error reading the alloc's secrets directory.

Common situations: Alloc directories migrated or restored from backups with wrong uid/gid; running the Nomad agent as a different user than originally; encrypted/readonly filesystem issues on the host data dir.

Related errors

• failed to write Vault token to disk: %w
• Failed to create task mount directory: %v
• No client configuration found for Vault cluster %s
• failed to write vault token to secrets dir: %v
• failed to write vault token: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a763978fe90c5824. Report an issue: GitHub.