XTLS/Xray-core · critical

parse server public key: not rsa

Error message

parse server public key: not rsa

What it means

x509.ParsePKIXPublicKey succeeded but returned a key that is not *rsa.PublicKey (for example an ECDSA or Ed25519 key). The xmc protocol requires RSA because the client encrypts the 16-byte shared secret and the password-suffixed verify token with rsa.EncryptPKCS1v15. Any non-RSA key cannot be used for this handshake.

Source

Thrown at transport/internet/finalmask/xmc/client.go:152

	)

	err = pkt.readFields(&serverId, &publicKey, &verifyToken)
	if err != nil {
		return fmt.Errorf("read encryption request fields: %w", err)
	}

	if !bytes.Equal(publicKey, c.rsaPublicKey) {
		return fmt.Errorf("server public key mismatch")
	}

	k, err := x509.ParsePKIXPublicKey(publicKey)
	if err != nil {
		return fmt.Errorf("parse server public key: %w", err)
	}

	rsaPublicKey, ok := k.(*rsa.PublicKey)
	if !ok {
		return fmt.Errorf("parse server public key: not rsa")
	}

	sharedSecret := make([]byte, 16)
	if _, err = rand.Read(sharedSecret); err != nil {
		return fmt.Errorf("generate shared secret: %w", err)
	}

	encryptedSharedSecret, err := rsa.EncryptPKCS1v15(rand.Reader, rsaPublicKey, sharedSecret)
	if err != nil {
		return fmt.Errorf("encrypt shared secret: %w", err)
	}

	verifyToken = append(verifyToken, []byte(c.password)...) // append pre-shared password

	encryptedVerifyToken, err := rsa.EncryptPKCS1v15(rand.Reader, rsaPublicKey, verifyToken)
	if err != nil {
		return fmt.Errorf("encrypt verify token: %w", err)
	}

View on GitHub (pinned to 7d214f8b09)

Solutions

1. Generate an RSA key (>= 2048 bits, e.g. crypto/rsa.GenerateKey(rand.Reader, 2048)) and export with x509.MarshalPKIXPublicKey
2. Replace the mis-typed key in Config.RsaPublicKey on the client and in the server's key config
3. Add a startup self-check: parse the configured key and assert the *rsa.PublicKey type before dialing

Example fix

// before
priv, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
der, _ := x509.MarshalPKIXPublicKey(&priv.PublicKey)
cfg.RsaPublicKey = der

// after
priv, _ := rsa.GenerateKey(rand.Reader, 2048)
der, _ := x509.MarshalPKIXPublicKey(&priv.PublicKey)
cfg.RsaPublicKey = der

k, err := x509.ParsePKIXPublicKey(cfg.RsaPublicKey)
if err != nil {
    return fmt.Errorf("config key unparsable: %w", err)
}
if _, ok := k.(*rsa.PublicKey); !ok {
    return fmt.Errorf("config key must be RSA, got %T", k)
}

func isRSAKey(b []byte) bool {
    k, err := x509.ParsePKIXPublicKey(b)
    if err != nil {
        return false
    }
    _, ok := k.(*rsa.PublicKey)
    return ok
}

When it happens

Trigger: Config.WrapConnClient connects to a server whose advertised rsa_public_key decodes to an ECDSA/Ed25519 PublicKey. The x509 parse succeeds, then the type assertion k.(*rsa.PublicKey) fails.

Common situations: An operator generated a modern Ed25519 or P-256 key with openssl and put it where the protocol requires RSA; or a key-generation script defaulted to ECDSA. Note that a PKCS#1 RSA key would instead fail at the ParsePKIX step (error 860), not here.

Related errors

• parse server public key: %w
• Failed to build REALITY config.
• encrypt shared secret: %w
• encrypt verify token: %w
• failed to compute mod inverse

AI-assisted analysis of XTLS/Xray-core@7d214f8b09 (2026-08-15). Data as JSON: /api/errors/9cef00badf98c177. Report an issue: GitHub.