hashicorp/nomad · error

failed to write vault token to secrets dir: %v

Error message

failed to write vault token to secrets dir: %v

What it means

writeToken persists the Vault token file for the task. On the upgrade path — when the task's private directory does not exist (pre-private-dir allocations) — it falls back to writing the token into the legacy secrets directory with os.WriteFile. If that write fails (permissions, disk full, path missing), this error is returned.

Source

Thrown at client/allocrunner/taskrunner/vault_hook.go:423

	// If the token cannot be renewed, it doesn't matter if the user set
	// allow_token_expiration or not, so override the requested behavior
	if !renewable {
		h.allowTokenExpiration = true
	}

	return token, leaseDuration, nil
}

// writeToken writes the given token to disk
func (h *vaultHook) writeToken(token string) error {
	// Handle upgrade path by first checking if the tasks private directory
	// exists. If it doesn't, this allocation probably existed before the
	// private directory was introduced, so keep using the secret directory to
	// prevent unnecessary errors during task recovery.
	if _, err := os.Stat(path.Dir(h.privateDirTokenPath)); os.IsNotExist(err) {
		if err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {
			return fmt.Errorf("failed to write vault token to secrets dir: %v", err)
		}
		return nil
	}

	if err := os.WriteFile(h.privateDirTokenPath, []byte(token), 0600); err != nil {
		return fmt.Errorf("failed to write vault token: %v", err)
	}
	if !h.vaultBlock.DisableFile {
		if err := os.WriteFile(h.secretsDirTokenPath, []byte(token), 0666); err != nil {
			return fmt.Errorf("failed to write vault token to secrets dir: %v", err)
		}
	}

	return nil
}

// withJitter returns when a token should be renewed given its leaseDuration
// and a randomizer to provide jitter.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check permissions/ownership of the alloc's secrets directory on the client host
2. Free disk space or resolve I/O errors on the Nomad data volume
3. Restart the allocation so the taskrunner recreates the private directory and normal write path
4. Ensure security tooling (SELinux/AppArmor) is not blocking writes to the Nomad alloc dir

Example fix

// host shell: fix perms on the alloc secrets dir
chmod u+rwX /var/lib/nomad/alloc/<alloc-id>/<task>/secrets

// before starting allocations, check the secrets dir is writable
st, err := os.Stat(secretsDir)
if err != nil || !st.IsDir() { /* fix dir */ }
f, err := os.OpenFile(filepath.Join(secretsDir, ".probe"), os.O_CREATE|os.O_WRONLY, 0666)
if err != nil { /* permissions problem */ }
f.Close(); os.Remove(filepath.Join(secretsDir, ".probe"))

if err := writeToken(token); err != nil {
    if strings.Contains(err.Error(), "failed to write vault token") {
        // inspect alloc dir perms/disk, then reschedule the alloc
        return rescheduleAlloc(allocID)
    }
    return err
}

When it happens

Trigger: Allocation created before the private token directory was introduced; os.Stat on the private dir returns IsNotExist, then os.WriteFile to h.secretsDirTokenPath fails due to filesystem permissions, a missing secrets dir, or I/O errors.

Common situations: Upgraded Nomad clients running old allocations; read-only or full data volumes; secrets dir permissions changed by external tooling (security hardening, SELinux).

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• failed to write vault token: %v
• plugin not executable
• Chmod(%v) failed: %w
• Couldn't change owner/group of %v to (uid: %v, gid: %v): %w
• unable to remove existing unix socket: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fbb129fc55842e7e. Report an issue: GitHub.