What causes "failed to load decoded CA Cert as PEM" in temporalio/temporal?

Base64-encode the PEM CA certificate file: base64 -w0 ca.pem, and put that string in caData

failed to load decoded CA Cert as PEM

Error message

failed to load decoded CA Cert as PEM

What it means

When CaData is supplied, ConfigureCassandraCluster base64-decodes it and appends the resulting bytes to an x509 CertPool with AppendCertsFromPEM. This error means the decoded bytes were not parseable as PEM certificate(s), so the root CA pool would be empty and TLS verification would fail. AppendCertsFromPEM returns false silently, and the plugin turns that into this error.

Source

Thrown at common/persistence/nosql/nosqlplugin/cassandra/gocql/client.go:122

		}

		if len(certBytes) > 0 {
			clientCert, err := tls.X509KeyPair(certBytes, keyBytes)
			if err != nil {
				return fmt.Errorf("unable to generate x509 key pair: %w", err)
			}

			cluster.SslOpts.Certificates = []tls.Certificate{clientCert}
		}

		if cfg.TLS.CaData != "" {
			cluster.SslOpts.RootCAs = x509.NewCertPool()
			pem, err := base64.StdEncoding.DecodeString(cfg.TLS.CaData)
			if err != nil {
				return fmt.Errorf("caData could not be decoded: %w", err)
			}
			if !cluster.SslOpts.RootCAs.AppendCertsFromPEM(pem) {
				return errors.New("failed to load decoded CA Cert as PEM")
			}
		}
	}

	if cfg.MaxConns > 0 {
		cluster.NumConns = cfg.MaxConns
	}

	cluster.ConnectTimeout = 10 * time.Second * debug.TimeoutMultiplier
	if cfg.ConnectTimeout > 0 {
		cluster.ConnectTimeout = cfg.ConnectTimeout
	}

	cluster.Timeout = cluster.ConnectTimeout
	if cfg.Timeout > 0 {
		cluster.Timeout = cfg.Timeout
	}

View on GitHub (pinned to bde624efd1)

Solutions

Base64-encode the PEM CA certificate file: base64 -w0 ca.pem, and put that string in caData
Verify with: echo "$caData" | base64 -d | openssl x509 -text -noout, confirming a CA certificate is printed
Ensure the file contains the CA cert(s) with PEM headers intact and no leading/trailing junk
If you only have DER format, convert first: openssl x509 -inform DER -in ca.der -out ca.pem

Example fix

// before
caData: "-----BEGIN CERTIFICATE-----\nMIIFaz..."   // raw PEM, not base64
// after (shell)
caData: "$(base64 -w0 ca.pem)"                         // base64 of PEM bytes

Defensive patterns

Strategy: validation

Validate before calling

caData, err := base64.StdEncoding.DecodeString(cfg.TLS.CaData)
if err != nil {
    return fmt.Errorf("caData is not valid base64: %w", err)
}
if !x509.NewCertPool().AppendCertsFromPEM(caData) {
    return errors.New("caData does not decode to PEM certificates")
}

Prevention

Generate caData with `base64 -w0 ca.pem` and verify with `echo $caData | base64 -d | openssl x509 -text`
Keep the PEM headers intact; never strip BEGIN/END lines
Confirm the encoded file is a CA certificate, not a leaf key or CSR
Watch for YAML line-wrapping/escaping corrupting long base64 strings

When it happens

Trigger: Calling NewCassandraCluster with cfg.TLS.CaData set to a non-empty value that is valid base64 but whose decoded bytes are not PEM-formatted certificates (or are a private key/CSR instead of a CA cert).

Common situations: Passing raw PEM text (-----BEGIN CERTIFICATE-----) as caData instead of base64-encoding it first; base64-encoding a DER binary when AppendCertsFromPEM needs PEM; including only the intermediate cert with formatting damage (CRLF stripped, extra whitespace); encoding the wrong file.

Related errors

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/26e584a873807994. Report an issue: GitHub.