nats-io/nats-server · error

ErrBadPublicKeyAlgorithm

ErrBadPublicKeyAlgorithm

Error message

unsupported public key algorithm

What it means

ErrBadPublicKeyAlgorithm is returned when, after extracting the private key's algorithm group from the Windows store, the value is neither "ECDSA" nor "RSA" (the default branch at certstore_windows.go:766). The library only knows how to rebuild public keys for those two algorithm families; keys of any other algorithm group cannot be mapped to a Go public key.

Source

Thrown at server/certstore/errors.go:39

	ErrStoreECDSASigningError = errors.New("unable to obtain ECDSA signature from store")

	// ErrNoPrivateKeyStoreRef represents an error getting a handle to a private key in store
	ErrNoPrivateKeyStoreRef = errors.New("unable to obtain private key handle from store")

	// ErrExtractingPrivateKeyMetadata represents a family of errors extracting metadata about the private key in store
	ErrExtractingPrivateKeyMetadata = errors.New("unable to extract private key metadata")

	// ErrExtractingECCPublicKey represents an error exporting ECC-type public key from store
	ErrExtractingECCPublicKey = errors.New("unable to extract ECC public key from store")

	// ErrExtractingRSAPublicKey represents an error exporting RSA-type public key from store
	ErrExtractingRSAPublicKey = errors.New("unable to extract RSA public key from store")

	// ErrExtractingPublicKey represents a general error exporting public key from store
	ErrExtractingPublicKey = errors.New("unable to extract public key from store")

	// ErrBadPublicKeyAlgorithm represents a bad or unsupported public key algorithm
	ErrBadPublicKeyAlgorithm = errors.New("unsupported public key algorithm")

	// ErrExtractPropertyFromKey represents a general failure to extract a metadata property field
	ErrExtractPropertyFromKey = errors.New("unable to extract property from key")

	// ErrBadECCCurveName represents an ECC signature curve name that is bad or unsupported
	ErrBadECCCurveName = errors.New("unsupported ECC curve name")

	// ErrFailedCertSearch represents not able to find certificate in store
	ErrFailedCertSearch = errors.New("unable to find certificate in store")

	// ErrFailedX509Extract represents not being able to extract x509 certificate from found cert in store
	ErrFailedX509Extract = errors.New("unable to extract x509 from certificate")

	// ErrBadMatchByType represents unknown CERT_MATCH_BY passed
	ErrBadMatchByType = errors.New("cert match by type not implemented")

	// ErrBadCertStore represents unknown CERT_STORE passed
	ErrBadCertStore = errors.New("cert store type not implemented")

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Reissue the certificate with an RSA or ECDSA key — other algorithm groups are not supported by TLS/this library.
2. Print/log the key's algorithm-group property to identify what the KSP actually reports.
3. Use a standard Microsoft KSP so the algorithm group is reported with the canonical "RSA"/"ECDSA" names.
4. Replace legacy DSA/DH certificates, which modern TLS stacks reject anyway.

Example fix

// before: DSA key -> ErrBadPublicKeyAlgorithm
// after: reissue with RSA
// openssl genrsa -out server.key 2048 && reissue cert

// before TLS setup, check the key algorithm group is one the library supports
alg, err := keyAlgorithmGroup(handle)
if err != nil || (alg != "RSA" && alg != "ECDSA") {
    return fmt.Errorf("key algorithm %q unsupported; reissue cert with RSA or ECDSA", alg)
}

func isSupportedKeyAlgGroup(alg string) bool {
    return alg == "RSA" || alg == "ECDSA"
}

if errors.Is(err, certstore.ErrBadPublicKeyAlgorithm) {
    return fmt.Errorf("replace the certificate with an RSA or ECDSA key: %w", err)
}

When it happens

Trigger: TLSConfig build on Windows with a store certificate whose NCRYPT_ALGORITHM_GROUP_PROPERTY returns an unrecognized value (not "ECDSA"/"RSA") — e.g. DH, DSA, or a vendor-specific algorithm group string — hitting `default: return nil, ErrBadPublicKeyAlgorithm` at :766.

Common situations: Certificates issued with legacy DSA or DH keys; vendor KSPs reporting nonstandard algorithm-group names; misprovisioned test certificates; keys migrated between KSPs with altered algorithm metadata.

Related errors

• ErrExtractingPrivateKeyMetadata
• ErrExtractingECCPublicKey
• ErrExtractingRSAPublicKey
• ErrExtractingPublicKey
• ErrBadRSAHashAlgorithm

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/7442c68304626d6d. Report an issue: GitHub.