argoproj/argo-workflows · error

failed to load system cert pool

Error message

failed to load system cert pool: %w

What it means

The SSO HTTP client builder clones http.DefaultTransport and loads the OS root certificate pool via x509.SystemCertPool() so that SSL_CERT_DIR/SSL_CERT_FILE are respected. If the system pool cannot be loaded (e.g. no root store available on the platform), the error is wrapped as 'failed to load system cert pool: %w' and SSO client creation fails. Per Go docs, SystemCertPool is unsupported on macOS and can fail on systems without any CA bundle.

Solutions

1. Install CA certificates in the container image: `apt-get install -y ca-certificates` (Debian) or the equivalent for your base image
2. Set SSL_CERT_FILE to a valid PEM bundle (e.g. the image's ca-certificates.crt) and ensure the file exists
3. Provide config.RootCA in the SSO ConfigMap and, if the platform still fails, add explicit fallback handling for SystemCertPool errors

Example fix

// Dockerfile
// before
FROM debian:slim
// after
FROM debian:slim
RUN apt-get update && apt-get install -y ca-certificates

if _, err := os.Stat(os.Getenv("SSL_CERT_FILE")); os.Getenv("SSL_CERT_FILE") != "" && err != nil {
    log.Printf("SSL_CERT_FILE points to missing file: %v", err)
}
if _, err := x509.SystemCertPool(); err != nil {
    log.Printf("system cert pool unavailable: %v", err)
}

func systemTrustAvailable() bool {
    _, err := x509.SystemCertPool()
    return err == nil
}

client, err := createHTTPClient(cfg)
if err != nil && strings.Contains(err.Error(), "failed to load system cert pool") {
    // install ca-certificates in the image or set SSL_CERT_FILE to a valid bundle
}

When it happens

Trigger: createHTTPClient (during SSO init at argo-server startup or in tests) calls x509.SystemCertPool() on a platform/manifest combination with no CA bundle: minimal containers lacking /etc/ssl/certs, musl images without ca-certificates installed, or macOS builds where it returns an error.

Common situations: Running argo-server in a distroless/slim image without the ca-certificates package; SSL_CERT_FILE/SSL_CERT_DIR pointing to nonexistent paths on some platforms; building a custom argo image that strips the trust store.

Related errors

• failed to append certificates from PEM string
• clientID empty
• expected an array, got
• failed to create HTTP client
• group name was not a string

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/37432b24ec73809c. Report an issue: GitHub.

Appendix: source

Thrown at server/auth/sso/clients.go:41

		}
	}

	return fmt.Sprintf("HTTPClientConfig{InsecureSkipVerify: %t, RootCA: %q (%d bytes)}",
		c.InsecureSkipVerify, rootCAPreview, rootCALen)
}

func createHTTPClient(config HTTPClientConfig) (*http.Client, error) {
	// Start with a copy of the default client
	httpClient := *http.DefaultClient

	// Clone the default transport and cast to *http.Transport
	defaultTransport := http.DefaultTransport.(*http.Transport)
	transport := defaultTransport.Clone()

	// Load system cert pool to respect env.SSL_CERT_DIR, env.SSL_CERT_FILE. macOS are not supported (https://pkg.go.dev/crypto/x509#SystemCertPool)
	rootCAs, err := x509.SystemCertPool()
	if err != nil {
		return nil, fmt.Errorf("failed to load system cert pool: %w", err)
	}

	// Set RootCAs if provided
	// Load root CA certificates from PEM string if defined
	if config.RootCA != "" {
		if ok := rootCAs.AppendCertsFromPEM([]byte(config.RootCA)); !ok {
			return nil, fmt.Errorf("failed to append certificates from PEM string")
		}
	}

	// Apply the custom TLS config to the cloned transport
	transport.TLSClientConfig = &tls.Config{
		InsecureSkipVerify: config.InsecureSkipVerify,
		RootCAs:            rootCAs,
	}

	// Use the modified transport in our client copy
	httpClient.Transport = transport

View on GitHub (pinned to 35bff19146)