hashicorp/nomad · critical

failed to fetch signed identities: %w

Error message

failed to fetch signed identities: %w

What it means

WIDMgr.Run wraps any failure from getInitialIdentities (which fetches freshly signed identities from the servers when the state DB has none or only expired ones) with 'failed to fetch signed identities: %w'. Run therefore aborts and the client cannot start with valid workload tokens.

Source

Thrown at client/widmgr/widmgr.go:130

//
// If an error is returned the identities could not be fetched and the renewal
// goroutine was not started.
func (m *WIDMgr) Run() error {
	if len(m.widSpecs) == 0 && len(m.defaultSignedIdentities) == 0 {
		m.logger.Debug("no workload identities to retrieve or renew")
		return nil
	}

	m.logger.Debug("retrieving and renewing workload identities", "num_identities", len(m.widSpecs))

	hasExpired, err := m.restoreStoredIdentities()
	if err != nil {
		m.logger.Warn("failed to get signed identities from state DB, refreshing from server",
			"error", err)
	}
	if hasExpired {
		if err := m.getInitialIdentities(); err != nil {
			return fmt.Errorf("failed to fetch signed identities: %w", err)
		}
	}

	go m.renew()

	return nil
}

// Get retrieves the latest signed identity or returns an error. It must be
// called after Run and does not block.
//
// For retrieving tokens which might be renewed callers should use Watch
// instead to avoid missing new tokens retrieved by Run between Get and Watch
// calls.
func (m *WIDMgr) Get(id structs.WIHandle) (*structs.SignedWorkloadIdentity, error) {
	token := m.get(id)
	if token == nil {
		// This is an error as every identity should have a token by the time Get

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %w error to find the root cause (RPC error vs signing rejection).
2. Verify connectivity between the Nomad client and servers.
3. Check server logs for Alloc.SignIdentities rejections and fix the underlying identity/policy issue.
4. Delete/repair the client state DB if stale identities cause repeated refresh failures.
5. Restart the agent after fixing connectivity so Run() can fetch identities again.

if err := widMgr.Run(); err != nil {
    if strings.Contains(err.Error(), "failed to fetch signed identities") {
        logger.Error("widmgr startup failed", "cause", err)
        // fix connectivity/policy, then restart the agent or task
    }
    return err
}

When it happens

Trigger: Run() detects hasExpired (no cached tokens in the state DB or all expired) and getInitialIdentities fails — typically because the upstream SignIdentities call errored (rejections, empty reply, RPC failure).

Common situations: Client node can't reach the servers; server rejects signing (version skew, policy); corrupted state DB yields stale tokens judged expired; first boot of an allocation with identities and no server connectivity.

Related errors

• failed to retrieve signed workload identity: %w
• %d/%d signing request was rejected: %v
• empty signed identity response
• expected %d signed identities but received %d
• no identities requested

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/644ab13aff929532. Report an issue: GitHub.