argoproj/argo-workflows · error

failed to append certificates from PEM string

Error message

failed to append certificates from PEM string

What it means

When the SSO config defines RootCA as a PEM string, createHTTPClient calls rootCAs.AppendCertsFromPEM, which returns false if no certificates could be decoded from the string. The function then returns 'failed to append certificates from PEM string'. AppendCertsFromPEM silently skips invalid entries, so this error means the entire RootCA value contained zero valid PEM certificates.

Source

Thrown at server/auth/sso/clients.go:48

func createHTTPClient(config HTTPClientConfig) (*http.Client, error) {
	// Start with a copy of the default client
	httpClient := *http.DefaultClient

	// Clone the default transport and cast to *http.Transport
	defaultTransport := http.DefaultTransport.(*http.Transport)
	transport := defaultTransport.Clone()

	// Load system cert pool to respect env.SSL_CERT_DIR, env.SSL_CERT_FILE. macOS are not supported (https://pkg.go.dev/crypto/x509#SystemCertPool)
	rootCAs, err := x509.SystemCertPool()
	if err != nil {
		return nil, fmt.Errorf("failed to load system cert pool: %w", err)
	}

	// Set RootCAs if provided
	// Load root CA certificates from PEM string if defined
	if config.RootCA != "" {
		if ok := rootCAs.AppendCertsFromPEM([]byte(config.RootCA)); !ok {
			return nil, fmt.Errorf("failed to append certificates from PEM string")
		}
	}

	// Apply the custom TLS config to the cloned transport
	transport.TLSClientConfig = &tls.Config{
		InsecureSkipVerify: config.InsecureSkipVerify,
		RootCAs:            rootCAs,
	}

	// Use the modified transport in our client copy
	httpClient.Transport = transport

	return &httpClient, nil
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Set rootCA to the literal PEM text: `cat ca.crt` output including BEGIN/END CERTIFICATE lines, in the argo-server SSO ConfigMap
2. If your value is base64, decode it once before storing: `base64 -d ca.b64 > ca.crt`, then paste ca.crt contents
3. Validate with `openssl x509 -in ca.crt -noout` and ensure the value is a certificate, not a key or CSR

Example fix

# ConfigMap
# before
rootCA: /etc/certs/ca.crt          # a path — wrong
# after
rootCA: |
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

if cfg.RootCA != "" && !x509.NewCertPool().AppendCertsFromPEM([]byte(cfg.RootCA)) {
    return errors.New("sso.rootCA does not contain any valid PEM certificates")
}

func isValidRootCAPEM(pemStr string) bool {
    pool := x509.NewCertPool()
    return pemStr == "" || pool.AppendCertsFromPEM([]byte(pemStr))
}

client, err := createHTTPClient(cfg)
if err != nil && strings.Contains(err.Error(), "failed to append certificates") {
    // log the HTTPClientConfig.String() preview and fix rootCA in the ConfigMap
}

When it happens

Trigger: sso.rootCA in the workflow-controller/argo-server ConfigMap is set but its value is not one or more '-----BEGIN CERTIFICATE-----' PEM blocks — e.g. it holds a private key, a raw base64 body without headers, an empty string with whitespace, or a URL/file path instead of the certificate contents.

Common situations: Users paste the CA file path rather than its contents into rootCA; the ConfigMap value was base64-encoded one time too many; line-wrapping corrupted the PEM; self-signed CA exported as PKCS#7/DER instead of PEM.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• failed to load system cert pool: %w
• issuer empty
• clientID empty
• no claim found for key: %v
• failure to create dynamic client: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/a1b8af44f2e8c21f. Report an issue: GitHub.