XTLS/Xray-core · critical

server public key mismatch

Error message

server public key mismatch

What it means

The client pins the server's RSA key: the public key bytes in the Encryption Request must equal the rsa_public_key configured locally, and a mismatch aborts the handshake. This is deliberate certificate-style pinning; it catches both man-in-the-middle substitution and simple key/config drift between client and server.

Source

Thrown at transport/internet/finalmask/xmc/client.go:142

	}

	if pkt.packetID != 0x01 {
		return fmt.Errorf("bad encrypt request packet id")
	}

	var (
		serverId    String
		publicKey   Bytes
		verifyToken Bytes
	)

	err = pkt.readFields(&serverId, &publicKey, &verifyToken)
	if err != nil {
		return fmt.Errorf("read encryption request fields: %w", err)
	}

	if !bytes.Equal(publicKey, c.rsaPublicKey) {
		return fmt.Errorf("server public key mismatch")
	}

	k, err := x509.ParsePKIXPublicKey(publicKey)
	if err != nil {
		return fmt.Errorf("parse server public key: %w", err)
	}

	rsaPublicKey, ok := k.(*rsa.PublicKey)
	if !ok {
		return fmt.Errorf("parse server public key: not rsa")
	}

	sharedSecret := make([]byte, 16)
	if _, err = rand.Read(sharedSecret); err != nil {
		return fmt.Errorf("generate shared secret: %w", err)
	}

	encryptedSharedSecret, err := rsa.EncryptPKCS1v15(rand.Reader, rsaPublicKey, sharedSecret)

View on GitHub (pinned to 7d214f8b09)

Solutions

1. Re-export the server's current public key exactly as the server sends it (PKIX DER) and update rsa_public_key on the client.
2. Ensure both sides load the same key pair and that no re-encoding (PEM wrappers, whitespace) altered the bytes.
3. If the key was not intentionally changed, treat the mismatch as a possible MITM and investigate before trusting.

Example fix

// before (stale key)
"rsaPublicKey": "MIIBIjANBgkq...old..."
// after (re-exported from server)
"rsaPublicKey": "MIIBIjANBgkq...new..."

pub, err := x509.ParsePKIXPublicKey(cfg.RsaPublicKey)
if err != nil || pub == nil {
	return errors.New("rsa_public_key is not valid PKIX DER")
}
if _, ok := pub.(*rsa.PublicKey); !ok {
	return errors.New("rsa_public_key is not an RSA key")
}

func isRSAPublicKeyDER(der []byte) bool {
	k, err := x509.ParsePKIXPublicKey(der)
	if err != nil {
		return false
	}
	_, ok := k.(*rsa.PublicKey)
	return ok
}

if err := cc.Handshake(); err != nil {
	if strings.Contains(err.Error(), "server public key mismatch") {
		// Key pinning failure: either rotate config from the server's current key,
		// or halt and investigate a possible MITM. Never auto-trust the new key.
		return fmt.Errorf("pinned key mismatch for %s: verify out-of-band before updating config", hostname)
	}
	return err
}

When it happens

Trigger: Server rotated or regenerated its RSA key pair while the client config still holds the old public key; client config pasted with a different server's key; an actual MITM presenting its own key. The byte-for-byte bytes.Equal comparison leaves no tolerance for encoding differences (e.g. PEM vs DER, re-encoded PKIX).

Common situations: Server restart with auto-generated keys; copying configs between environments; DER-vs-PEM encoding mixups; key regenerated by an orchestrator on redeploy.

Related errors

• empty rsa public key
• select profile: %w
• derive minecraft rsa key: %w
• marshal minecraft rsa public key: %w
• empty profiles

AI-assisted analysis of XTLS/Xray-core@7d214f8b09 (2026-08-15). Data as JSON: /api/errors/04cf75323beee79a. Report an issue: GitHub.