Netflix/chaosmonkey · critical

tls.X509KeyPair failed

Error message

tls.X509KeyPair failed

What it means

spinnaker.getClient converted the PKCS#12 data to PEM blocks and then called tls.X509KeyPair on the first block (assumed cert) and last block (assumed private key). This error means the pair failed to load: the PEM blocks do not contain a valid certificate/private key, they don't match, or the p12 contained additional blocks so the first/last heuristic picked the wrong ones.

Source

Thrown at spinnaker/spinnaker.go:79

type spinnakerInstance struct {
	Name string
}

// getClient takes PKCS#12 data (encrypted cert data in .p12 format) and the
// password for the encrypted cert, and returns an http client that does TLS client auth
func getClient(pfxData []byte, password string) (*http.Client, error) {
	blocks, err := pkcs12.ToPEM(pfxData, password)
	if err != nil {
		return nil, errors.Wrap(err, "pkcs.ToPEM failed")
	}

	// The first block is the cert and the last block is the private key
	certPEMBlock := pem.EncodeToMemory(blocks[0])
	keyPEMBlock := pem.EncodeToMemory(blocks[len(blocks)-1])

	cert, err := tls.X509KeyPair(certPEMBlock, keyPEMBlock)
	if err != nil {
		return nil, errors.Wrap(err, "tls.X509KeyPair failed")
	}

	tlsConfig := &tls.Config{
		Certificates: []tls.Certificate{cert},
	}
	transport := &http.Transport{TLSClientConfig: tlsConfig}
	return &http.Client{Transport: transport}, nil
}

// getClientX509 takes X509 data (Public and Private keys) and the
// and returns an http client that does TLS client auth
func getClientX509(x509Cert, x509Key string) (*http.Client, error) {
	cert, err := tls.LoadX509KeyPair(x509Cert, x509Key)
	if err != nil {
		return nil, errors.Wrap(err, "tls.X509KeyPair failed")
	}
	tlsConfig := &tls.Config{
		Certificates:       []tls.Certificate{cert},

View on GitHub (pinned to eaa28fb761)

Solutions

  1. Re-export the .p12 containing exactly one certificate and its matching private key, with the leaf cert first (e.g. `openssl pkcs12 -export -inkey key.pem -in cert.pem -certfile fullchain.pem` and verify order, or exclude the chain)
  2. Inspect the p12 contents (`openssl pkcs12 -in cert.p12 -nokeys` / `-nocerts`) and confirm block order: first = leaf cert, last = private key
  3. Verify the private key in the p12 matches the certificate (compare public keys via openssl x509/pkey modulus)
  4. As a workaround, use the x509 path instead: pass x509Cert/x509Key PEM file paths to spinnaker.New

Example fix

// before
openssl pkcs12 -export -out s.p12 -in fullchain.pem -inkey key.pem   # CA-first p12; blocks[0] is not the leaf cert
// after
openssl pkcs12 -export -out s.p12 -in leaf-cert.pem -inkey key.pem   # leaf cert first, key last
Defensive patterns

Strategy: validation

Validate before calling

blocks, err := pkcs12.ToPEM(pfx, password)
if err != nil {
	return err
}
if len(blocks) < 2 {
	return fmt.Errorf("p12 must contain at least a cert and a key, got %d blocks", len(blocks))
}
// Mirror the library's first=cert, last=key heuristic before calling New
certPEM := pem.EncodeToMemory(blocks[0])
keyPEM := pem.EncodeToMemory(blocks[len(blocks)-1])
if _, err := tls.X509KeyPair(certPEM, keyPEM); err != nil {
	return fmt.Errorf("p12 block layout incompatible: %w", err)
}

Try / catch

sp, err := spinnaker.New(endpoint, certPath, password, "", "", user)
if err != nil && strings.Contains(err.Error(), "tls.X509KeyPair failed") {
	return fmt.Errorf("p12 must have the leaf cert first and private key last; re-export it: %w", err)
}

Prevention

When it happens

Trigger: Calling spinnaker.New with a .p12 whose block layout isn't cert-first/key-last (e.g. p12 includes CA chain blocks or multiple certs), or a p12 that decodes but whose first/last blocks aren't a matching cert+key (e.g. key encrypted differently or blocks reordered).

Common situations: p12 exported with the full chain so blocks[0] is an intermediate CA, not the leaf cert; p12 with extra attributes producing >2 blocks; corrupted re-export where cert and key don't correspond.

Understand the failure class

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/85c5a8598919c0f9. Report an issue: GitHub.