{"record":{"id":"c3f9e48e8dc2fe50","repo":"hashicorp/nomad","slug":"could-not-resolve-node-secret-w","errorCode":null,"errorMessage":"could not resolve node secret: %w","messagePattern":"could not resolve node secret: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/auth/auth.go","lineNumber":169,"sourceCode":"\t\t}\n\n\t\targs.SetIdentity(&structs.AuthenticatedIdentity{Claims: claims})\n\t\treturn nil\n\n\tcase errors.Is(err, structs.ErrTokenNotFound):\n\t\t// Check if the secret ID is the leader's secret ID, in which case treat\n\t\t// it as a management token.\n\t\tleaderAcl := s.getLeaderACL()\n\t\tif leaderAcl != \"\" && secretID == leaderAcl {\n\t\t\taclToken = structs.LeaderACLToken\n\t\t\tbreak\n\t\t} else {\n\t\t\t// Otherwise, see if the secret ID belongs to a node. We should\n\t\t\t// reach this point only on first connection.\n\t\t\tnode, err := s.getState().NodeBySecretID(nil, secretID)\n\t\t\tif err != nil {\n\t\t\t\t// this is a go-memdb error; shouldn't happen\n\t\t\t\treturn fmt.Errorf(\"could not resolve node secret: %w\", err)\n\t\t\t}\n\t\t\tif node != nil {\n\t\t\t\targs.SetIdentity(&structs.AuthenticatedIdentity{ClientID: node.ID})\n\t\t\t\treturn nil\n\t\t\t}\n\t\t}\n\n\t\t// we were passed a bogus token so we'll return an error, but we'll also\n\t\t// want to capture the IP for metrics\n\t\tremoteIP, err := ctx.GetRemoteIP()\n\t\tif err != nil {\n\t\t\ts.logger.Error(\"could not determine remote address\", \"error\", err)\n\t\t}\n\t\targs.SetIdentity(&structs.AuthenticatedIdentity{RemoteIP: remoteIP})\n\t\treturn structs.ErrPermissionDenied\n\n\tdefault: // any other error\n\t\treturn fmt.Errorf(\"could not resolve user: %w\", err)","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/auth/auth.go#L151-L187","documentation":"During Authenticate, when the secret ID is not an ACL token, the server looks it up as a node secret ID via NodeBySecretID. If that state-store query itself errors (a go-memdb error that 'shouldn't happen'), it is wrapped as 'could not resolve node secret'. This signals an internal store failure, not an invalid credential.","triggerScenarios":"An RPC authentication attempt whose bearer token is not resolvable as an ACL token, and the subsequent NodeBySecretID store lookup returns an error (store corruption, index failure, transient state store issue).","commonSituations":"Corrupted Nomad state store (boltdb/memdb) on the server; errors during restore from snapshot; internal Raft/state inconsistency after a crash.","solutions":["Inspect the wrapped error (`%w`) in server logs to see the underlying memdb/boltdb cause","Restart the server process to rebuild in-memory indexes from the state store","Verify the state store integrity; restore from a recent snapshot if corruption is confirmed","Retry the RPC after the server stabilizes — this is not an auth failure, so the token itself is fine"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"could not resolve node secret\") {\n    // internal store error, not auth failure: backoff and retry RPC\n    time.Sleep(backoff)\n    retry()\n}","preventionTips":["Monitor server state-store health and snapshot/restore procedures","Restart servers cleanly rather than killing during compaction","Keep regular Nomad snapshots for recovery from corruption"],"tags":["nomad","authentication","state-store","rpc"],"backgroundTag":"state-store-error","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}