hashicorp/nomad · error

failed to write Vault token to disk: %w

Error message

failed to write Vault token to disk: %w

What it means

When a Vault token renewal fails, handleRenewalFailure derives a brand-new token and persists it via writeToken before signaling/restarting per the vault block's change_mode. If writing the new token to the task's secrets directory fails (write error, missing directory, permissions), the recovery aborts with this wrapped error and the task is ultimately killed or evented.

Source

Thrown at client/allocrunner/taskrunner/vault_hook.go:263

						structs.NewTaskEvent(structs.TaskKilling).
							SetFailsTask().
							SetDisplayMessage(fmt.Sprintf("Vault: %v", err)))

					return
				}
			}
		}
	}
}

// handleRenewalFailure attempts to get a new Vault token and triggers any change_mode
func (h *vaultHook) handleRenewalFailure(ctx context.Context) (string, time.Duration, error) {
	token, duration, err := h.deriveVaultToken(ctx)
	if err != nil {
		return "", 0, err
	}
	if err := h.writeToken(token); err != nil {
		return "", 0, fmt.Errorf("failed to write Vault token to disk: %w", err)
	}

	var event *structs.TaskEvent
	switch h.vaultBlock.ChangeMode {
	case structs.VaultChangeModeSignal:
		s, err := signals.Parse(h.vaultBlock.ChangeSignal)
		if err != nil {
			return "", 0, fmt.Errorf("failed to parse signal: %w", err)
		}

		event := structs.NewTaskEvent(structs.TaskSignaling).
			SetTaskSignal(s).SetDisplayMessage("Vault: new Vault token acquired")
		if err := h.lifecycle.Signal(event, h.vaultBlock.ChangeSignal); err != nil {
			return "", 0, fmt.Errorf("failed to send signal: %w", err)
		}
	case structs.VaultChangeModeRestart:
		event = structs.NewTaskEvent(structs.TaskRestartSignal).
			SetDisplayMessage("Vault: new Vault token acquired")

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Free disk space on the client host and restart the task
2. Verify the task's secrets directory exists and is writable by the Nomad agent user
3. Check vault_hook/token write logs for the inner error wrapped by this message
4. Ensure consistent agent user/permissions on the alloc directories

// ensure secrets dir is writable before relying on token persistence
info, err := os.Stat(taskDir.SecretsDir)
if err != nil || !info.IsDir() {
    return fmt.Errorf("secrets dir missing/not a dir: %w", err)
}
if err := os.WriteFile(filepath.Join(taskDir.SecretsDir, ".probe"), []byte("x"), 0o600); err != nil {
    return fmt.Errorf("secrets dir not writable: %w", err)
}

if err := h.writeToken(token); err != nil {
    return "", 0, fmt.Errorf("failed to write Vault token to disk: %w (verify disk space and secrets dir permissions)", err)
}

When it happens

Trigger: Vault renewal expired/failed and the fallback re-derivation succeeded, but writeToken could not persist the token — e.g. the task's secrets dir was removed, disk full, or permissions were lost mid-task.

Common situations: Disk full on the client host; alloc directory garbage-collected while the task ran; permission changes from running under a different user; NFS/readonly mount for the alloc dir.

Related errors

• failed to recover vault token from %s: %v
• No client configuration found for Vault cluster %s
• failed to remove alloc dir %q: %w
• Failed to make the alloc directory %v: %w
• Failed to lookup nobody user: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/4d098757311834d2. Report an issue: GitHub.