hashicorp/nomad · error

failed to write vault token

Error message

failed to write vault token: %v

What it means

writeToken writes the token to the task's private directory path (h.privateDirTokenPath, mode 0600) as the primary storage. If os.WriteFile fails there — because the private dir doesn't exist, is unwritable, or the disk has errors — the hook returns this error and the task does not receive a Vault token.

Solutions

1. Ensure the task's private directory exists and is writable by the Nomad client user
2. Check host disk space and filesystem health (dmesg for read-only remounts)
3. Restart the allocation to have the taskrunner recreate its directory structure
4. Audit cron/cleanup jobs that might delete files under the Nomad alloc dir

Example fix

// host shell
# check disk and alloc dir
 df -h /var/lib/nomad
 ls -la /var/lib/nomad/alloc/<alloc-id>/<task>/private

// verify private dir exists and is writable before token derivation
if _, err := os.Stat(privateDir); err != nil {
    os.MkdirAll(privateDir, 0700)
}
probe := filepath.Join(privateDir, ".probe")
if err := os.WriteFile(probe, nil, 0600); err != nil { /* fix fs */ }
os.Remove(probe)

if err := writeToken(token); err != nil {
    if strings.Contains(err.Error(), "failed to write vault token:") &&
       !strings.Contains(err.Error(), "secrets dir") {
        checkDiskAndPerms(privateDirPath)
        return retryAfterFix()
    }
    return err
}

When it happens

Trigger: The private directory exists (upgrade-path check passed) but writing the token file to it fails: missing parent directory, wrong ownership/permissions, read-only filesystem, or disk full.

Common situations: Task private dir cleaned up externally while alloc runs; Nomad data volume full; container/VM filesystem gone read-only; tmpwatch-like cleaners deleting alloc directories.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• failed to write vault token to secrets dir
• Chmod( ) failed
• could not read path from keystore
• Couldn't change owner/group of
• error chmoding file

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b029caa25e7fbe5a. Report an issue: GitHub.

Appendix: source

Thrown at client/allocrunner/taskrunner/vault_hook.go:429

	return token, leaseDuration, nil
}

// writeToken writes the given token to disk
func (h *vaultHook) writeToken(token string) error {
	// Handle upgrade path by first checking if the tasks private directory
	// exists. If it doesn't, this allocation probably existed before the
	// private directory was introduced, so keep using the secret directory to
	// prevent unnecessary errors during task recovery.
	if _, err := os.Stat(path.Dir(h.privateDirTokenPath)); os.IsNotExist(err) {
		if err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {
			return fmt.Errorf("failed to write vault token to secrets dir: %v", err)
		}
		return nil
	}

	if err := os.WriteFile(h.privateDirTokenPath, []byte(token), 0600); err != nil {
		return fmt.Errorf("failed to write vault token: %v", err)
	}
	if !h.vaultBlock.DisableFile {
		if err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {
			return fmt.Errorf("failed to write vault token to secrets dir: %v", err)
		}
	}

	return nil
}

// withJitter returns when a token should be renewed given its leaseDuration
// and a randomizer to provide jitter.
//
// Leases < 1m will not use jitter.
func withJitter(leaseDuration time.Duration) time.Duration {
	// Start trying to renew at half the lease duration to allow ample time
	// for latency and retries.
	renew := leaseDuration / 2

View on GitHub (pinned to 482b49bf1a)