hashicorp/nomad · error

failed to write vault token: %v

Error message

failed to write vault token: %v

What it means

writeToken writes the token to the task's private directory path (h.privateDirTokenPath, mode 0600) as the primary storage. If os.WriteFile fails there — because the private dir doesn't exist, is unwritable, or the disk has errors — the hook returns this error and the task does not receive a Vault token.

Source

Thrown at client/allocrunner/taskrunner/vault_hook.go:429

	return token, leaseDuration, nil
}

// writeToken writes the given token to disk
func (h *vaultHook) writeToken(token string) error {
	// Handle upgrade path by first checking if the tasks private directory
	// exists. If it doesn't, this allocation probably existed before the
	// private directory was introduced, so keep using the secret directory to
	// prevent unnecessary errors during task recovery.
	if _, err := os.Stat(path.Dir(h.privateDirTokenPath)); os.IsNotExist(err) {
		if err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {
			return fmt.Errorf("failed to write vault token to secrets dir: %v", err)
		}
		return nil
	}

	if err := os.WriteFile(h.privateDirTokenPath, []byte(token), 0600); err != nil {
		return fmt.Errorf("failed to write vault token: %v", err)
	}
	if !h.vaultBlock.DisableFile {
		if err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {
			return fmt.Errorf("failed to write vault token to secrets dir: %v", err)
		}
	}

	return nil
}

// withJitter returns when a token should be renewed given its leaseDuration
// and a randomizer to provide jitter.
//
// Leases < 1m will not use jitter.
func withJitter(leaseDuration time.Duration) time.Duration {
	// Start trying to renew at half the lease duration to allow ample time
	// for latency and retries.
	renew := leaseDuration / 2

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Ensure the task's private directory exists and is writable by the Nomad client user
2. Check host disk space and filesystem health (dmesg for read-only remounts)
3. Restart the allocation to have the taskrunner recreate its directory structure
4. Audit cron/cleanup jobs that might delete files under the Nomad alloc dir

Example fix

// host shell
# check disk and alloc dir
 df -h /var/lib/nomad
 ls -la /var/lib/nomad/alloc/<alloc-id>/<task>/private

// verify private dir exists and is writable before token derivation
if _, err := os.Stat(privateDir); err != nil {
    os.MkdirAll(privateDir, 0700)
}
probe := filepath.Join(privateDir, ".probe")
if err := os.WriteFile(probe, nil, 0600); err != nil { /* fix fs */ }
os.Remove(probe)

if err := writeToken(token); err != nil {
    if strings.Contains(err.Error(), "failed to write vault token:") &&
       !strings.Contains(err.Error(), "secrets dir") {
        checkDiskAndPerms(privateDirPath)
        return retryAfterFix()
    }
    return err
}

When it happens

Trigger: The private directory exists (upgrade-path check passed) but writing the token file to it fails: missing parent directory, wrong ownership/permissions, read-only filesystem, or disk full.

Common situations: Task private dir cleaned up externally while alloc runs; Nomad data volume full; container/VM filesystem gone read-only; tmpwatch-like cleaners deleting alloc directories.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• failed to write vault token to secrets dir: %v
• plugin not executable
• Chmod(%v) failed: %w
• Couldn't change owner/group of %v to (uid: %v, gid: %v): %w
• unable to remove existing unix socket: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b029caa25e7fbe5a. Report an issue: GitHub.