{"record":{"id":"644ab13aff929532","repo":"hashicorp/nomad","slug":"failed-to-fetch-signed-identities-w","errorCode":null,"errorMessage":"failed to fetch signed identities: %w","messagePattern":"failed to fetch signed identities: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/widmgr/widmgr.go","lineNumber":130,"sourceCode":"//\n// If an error is returned the identities could not be fetched and the renewal\n// goroutine was not started.\nfunc (m *WIDMgr) Run() error {\n\tif len(m.widSpecs) == 0 && len(m.defaultSignedIdentities) == 0 {\n\t\tm.logger.Debug(\"no workload identities to retrieve or renew\")\n\t\treturn nil\n\t}\n\n\tm.logger.Debug(\"retrieving and renewing workload identities\", \"num_identities\", len(m.widSpecs))\n\n\thasExpired, err := m.restoreStoredIdentities()\n\tif err != nil {\n\t\tm.logger.Warn(\"failed to get signed identities from state DB, refreshing from server\",\n\t\t\t\"error\", err)\n\t}\n\tif hasExpired {\n\t\tif err := m.getInitialIdentities(); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to fetch signed identities: %w\", err)\n\t\t}\n\t}\n\n\tgo m.renew()\n\n\treturn nil\n}\n\n// Get retrieves the latest signed identity or returns an error. It must be\n// called after Run and does not block.\n//\n// For retrieving tokens which might be renewed callers should use Watch\n// instead to avoid missing new tokens retrieved by Run between Get and Watch\n// calls.\nfunc (m *WIDMgr) Get(id structs.WIHandle) (*structs.SignedWorkloadIdentity, error) {\n\ttoken := m.get(id)\n\tif token == nil {\n\t\t// This is an error as every identity should have a token by the time Get","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/widmgr/widmgr.go#L112-L148","documentation":"WIDMgr.Run wraps any failure from getInitialIdentities (which fetches freshly signed identities from the servers when the state DB has none or only expired ones) with 'failed to fetch signed identities: %w'. Run therefore aborts and the client cannot start with valid workload tokens.","triggerScenarios":"Run() detects hasExpired (no cached tokens in the state DB or all expired) and getInitialIdentities fails — typically because the upstream SignIdentities call errored (rejections, empty reply, RPC failure).","commonSituations":"Client node can't reach the servers; server rejects signing (version skew, policy); corrupted state DB yields stale tokens judged expired; first boot of an allocation with identities and no server connectivity.","solutions":["Inspect the wrapped %w error to find the root cause (RPC error vs signing rejection).","Verify connectivity between the Nomad client and servers.","Check server logs for Alloc.SignIdentities rejections and fix the underlying identity/policy issue.","Delete/repair the client state DB if stale identities cause repeated refresh failures.","Restart the agent after fixing connectivity so Run() can fetch identities again."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := widMgr.Run(); err != nil {\n    if strings.Contains(err.Error(), \"failed to fetch signed identities\") {\n        logger.Error(\"widmgr startup failed\", \"cause\", err)\n        // fix connectivity/policy, then restart the agent or task\n    }\n    return err\n}","preventionTips":["Ensure server reachability before client startup","Check state DB health; rebuild if tokens are persistently stale","Monitor signing rejections on servers"],"tags":["workload-identity","startup","rpc","nomad"],"backgroundTag":"identity-signing-rejected","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}