What causes "unsupported RSA hash algorithm" in nats-io/nats-server?

Use a supported hash: crypto.SHA256, crypto.SHA384, or crypto.SHA512 when calling Sign (or let crypto/tls pick one).

ErrBadRSAHashAlgorithm

ErrBadRSAHashAlgorithm

Error message

unsupported RSA hash algorithm

What it means

ErrBadRSAHashAlgorithm is returned by the Windows certificate-store signer when the requested crypto.Hash has no entry in the winAlgIDs lookup table mapping Go hash functions to Windows CNG algorithm identifiers. The library only supports a fixed set of RSA hash algorithms (e.g. SHA-256/384/512) when signing via the Windows certificate store; anything else cannot be mapped to an BCrypt algorithm ID. It is thrown before any Windows API call is made, so the certificate is never contacted.

Source

Thrown at server/certstore/errors.go:12

package certstore

import (
	"errors"
)

var (
	// ErrBadCryptoStoreProvider represents inablity to establish link with a certificate store
	ErrBadCryptoStoreProvider = errors.New("unable to open certificate store or store not available")

	// ErrBadRSAHashAlgorithm represents a bad or unsupported RSA hash algorithm
	ErrBadRSAHashAlgorithm = errors.New("unsupported RSA hash algorithm")

	// ErrBadSigningAlgorithm represents a bad or unsupported signing algorithm
	ErrBadSigningAlgorithm = errors.New("unsupported signing algorithm")

	// ErrStoreRSASigningError represents an error returned from store during RSA signature
	ErrStoreRSASigningError = errors.New("unable to obtain RSA signature from store")

	// ErrStoreECDSASigningError represents an error returned from store during ECDSA signature
	ErrStoreECDSASigningError = errors.New("unable to obtain ECDSA signature from store")

	// ErrNoPrivateKeyStoreRef represents an error getting a handle to a private key in store
	ErrNoPrivateKeyStoreRef = errors.New("unable to obtain private key handle from store")

	// ErrExtractingPrivateKeyMetadata represents a family of errors extracting metadata about the private key in store
	ErrExtractingPrivateKeyMetadata = errors.New("unable to extract private key metadata")

	// ErrExtractingECCPublicKey represents an error exporting ECC-type public key from store
	ErrExtractingECCPublicKey = errors.New("unable to extract ECC public key from store")

View on GitHub (pinned to 3a66a489d2)

Solutions

Use a supported hash: crypto.SHA256, crypto.SHA384, or crypto.SHA512 when calling Sign (or let crypto/tls pick one).
If you pre-hashed the data, still pass the corresponding crypto.Hash constant in opts rather than crypto.Hash(0).
Check the winAlgIDs map in server/certstore/certstore_windows.go to see exactly which hash functions the build supports.
If you need an unmapped algorithm, add the CNG algorithm ID to winAlgIDs or hash with a supported algorithm instead.

Example fix

// before
sig, err := signer.Sign(rand, digest, crypto.SHA1)
// after
sig, err := signer.Sign(rand, digest, crypto.SHA256)

Defensive patterns

Strategy: validation

Validate before calling

// before signing, ensure the hash is one the store supports
switch opts.(type) {
}
// map check (mirrors winAlgIDs)
var supported = map[crypto.Hash]bool{crypto.SHA256: true, crypto.SHA384: true, crypto.SHA512: true}
if hf != 0 && !supported[hf] {
    return fmt.Errorf("hash %v not supported by Windows cert store", hf)
}

Type guard

func isSupportedRSAHash(hf crypto.Hash) bool {
    switch hf {
    case crypto.SHA256, crypto.SHA384, crypto.SHA512:
        return true
    }
    return false
}

Try / catch

sig, err := signer.Sign(rand, digest, opts)
if errors.Is(err, certstore.ErrBadRSAHashAlgorithm) {
    // fall back to SHA-256
    sig, err = signer.Sign(rand, digest, crypto.SHA256)
}

Prevention

Always pass a concrete crypto.Hash (SHA-256/384/512) to Sign; never crypto.Hash(0).
Let crypto/tls drive signing so standard hash options are chosen automatically.
Check winAlgIDs in server/certstore before using unusual hashes on Windows.

When it happens

Trigger: Calling the certificate-store signing path (via TLSConfig with a store-backed private key on Windows) with a Signer_opts value whose crypto.Hash is not in winAlgIDs — e.g. crypto.MD5, crypto.SHA1 if excluded, or crypto.Hash(0) (no hash) — passed to the signer's Sign method, hitting the `algID, ok := winAlgIDs[hf]; if !ok` check at certstore_windows.go:549.

Common situations: Passing crypto.Hash(0) because the caller hashed the data itself and forgot opts; using a legacy hash like MD5/SHA-1 in custom signing code; wiring the store signer into crypto/tls with unusual signature algorithms; running custom hash enums on Windows where the CNG mapping table lacks an entry.

Related errors

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/0f88e444fb9be31d. Report an issue: GitHub.