hashicorp/nomad · error

could not resolve node secret: %w

Error message

could not resolve node secret: %w

What it means

During Authenticate, when the secret ID is not an ACL token, the server looks it up as a node secret ID via NodeBySecretID. If that state-store query itself errors (a go-memdb error that 'shouldn't happen'), it is wrapped as 'could not resolve node secret'. This signals an internal store failure, not an invalid credential.

Source

Thrown at nomad/auth/auth.go:169

		}

		args.SetIdentity(&structs.AuthenticatedIdentity{Claims: claims})
		return nil

	case errors.Is(err, structs.ErrTokenNotFound):
		// Check if the secret ID is the leader's secret ID, in which case treat
		// it as a management token.
		leaderAcl := s.getLeaderACL()
		if leaderAcl != "" && secretID == leaderAcl {
			aclToken = structs.LeaderACLToken
			break
		} else {
			// Otherwise, see if the secret ID belongs to a node. We should
			// reach this point only on first connection.
			node, err := s.getState().NodeBySecretID(nil, secretID)
			if err != nil {
				// this is a go-memdb error; shouldn't happen
				return fmt.Errorf("could not resolve node secret: %w", err)
			}
			if node != nil {
				args.SetIdentity(&structs.AuthenticatedIdentity{ClientID: node.ID})
				return nil
			}
		}

		// we were passed a bogus token so we'll return an error, but we'll also
		// want to capture the IP for metrics
		remoteIP, err := ctx.GetRemoteIP()
		if err != nil {
			s.logger.Error("could not determine remote address", "error", err)
		}
		args.SetIdentity(&structs.AuthenticatedIdentity{RemoteIP: remoteIP})
		return structs.ErrPermissionDenied

	default: // any other error
		return fmt.Errorf("could not resolve user: %w", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped error (`%w`) in server logs to see the underlying memdb/boltdb cause
2. Restart the server process to rebuild in-memory indexes from the state store
3. Verify the state store integrity; restore from a recent snapshot if corruption is confirmed
4. Retry the RPC after the server stabilizes — this is not an auth failure, so the token itself is fine

if strings.Contains(err.Error(), "could not resolve node secret") {
    // internal store error, not auth failure: backoff and retry RPC
    time.Sleep(backoff)
    retry()
}

When it happens

Trigger: An RPC authentication attempt whose bearer token is not resolvable as an ACL token, and the subsequent NodeBySecretID store lookup returns an error (store corruption, index failure, transient state store issue).

Common situations: Corrupted Nomad state store (boltdb/memdb) on the server; errors during restore from snapshot; internal Raft/state inconsistency after a crash.

Related errors

• could not resolve user: %w
• could not place volume %q: %w
• job %q in namespace %q not found
• error parsing: root should be an object
• cannot specify Accessor ID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c3f9e48e8dc2fe50. Report an issue: GitHub.