argoproj/argo-workflows · error

failed to load system cert pool: %w

Error message

failed to load system cert pool: %w

What it means

The SSO HTTP client builder clones http.DefaultTransport and loads the OS root certificate pool via x509.SystemCertPool() so that SSL_CERT_DIR/SSL_CERT_FILE are respected. If the system pool cannot be loaded (e.g. no root store available on the platform), the error is wrapped as 'failed to load system cert pool: %w' and SSO client creation fails. Per Go docs, SystemCertPool is unsupported on macOS and can fail on systems without any CA bundle.

Source

Thrown at server/auth/sso/clients.go:41

		}
	}

	return fmt.Sprintf("HTTPClientConfig{InsecureSkipVerify: %t, RootCA: %q (%d bytes)}",
		c.InsecureSkipVerify, rootCAPreview, rootCALen)
}

func createHTTPClient(config HTTPClientConfig) (*http.Client, error) {
	// Start with a copy of the default client
	httpClient := *http.DefaultClient

	// Clone the default transport and cast to *http.Transport
	defaultTransport := http.DefaultTransport.(*http.Transport)
	transport := defaultTransport.Clone()

	// Load system cert pool to respect env.SSL_CERT_DIR, env.SSL_CERT_FILE. macOS are not supported (https://pkg.go.dev/crypto/x509#SystemCertPool)
	rootCAs, err := x509.SystemCertPool()
	if err != nil {
		return nil, fmt.Errorf("failed to load system cert pool: %w", err)
	}

	// Set RootCAs if provided
	// Load root CA certificates from PEM string if defined
	if config.RootCA != "" {
		if ok := rootCAs.AppendCertsFromPEM([]byte(config.RootCA)); !ok {
			return nil, fmt.Errorf("failed to append certificates from PEM string")
		}
	}

	// Apply the custom TLS config to the cloned transport
	transport.TLSClientConfig = &tls.Config{
		InsecureSkipVerify: config.InsecureSkipVerify,
		RootCAs:            rootCAs,
	}

	// Use the modified transport in our client copy
	httpClient.Transport = transport

View on GitHub (pinned to 35bff19146)

Solutions

1. Install CA certificates in the container image: `apt-get install -y ca-certificates` (Debian) or the equivalent for your base image
2. Set SSL_CERT_FILE to a valid PEM bundle (e.g. the image's ca-certificates.crt) and ensure the file exists
3. Provide config.RootCA in the SSO ConfigMap and, if the platform still fails, add explicit fallback handling for SystemCertPool errors

Example fix

// Dockerfile
// before
FROM debian:slim
// after
FROM debian:slim
RUN apt-get update && apt-get install -y ca-certificates

if _, err := os.Stat(os.Getenv("SSL_CERT_FILE")); os.Getenv("SSL_CERT_FILE") != "" && err != nil {
    log.Printf("SSL_CERT_FILE points to missing file: %v", err)
}
if _, err := x509.SystemCertPool(); err != nil {
    log.Printf("system cert pool unavailable: %v", err)
}

func systemTrustAvailable() bool {
    _, err := x509.SystemCertPool()
    return err == nil
}

client, err := createHTTPClient(cfg)
if err != nil && strings.Contains(err.Error(), "failed to load system cert pool") {
    // install ca-certificates in the image or set SSL_CERT_FILE to a valid bundle
}

When it happens

Trigger: createHTTPClient (during SSO init at argo-server startup or in tests) calls x509.SystemCertPool() on a platform/manifest combination with no CA bundle: minimal containers lacking /etc/ssl/certs, musl images without ca-certificates installed, or macOS builds where it returns an error.

Common situations: Running argo-server in a distroless/slim image without the ca-certificates package; SSL_CERT_FILE/SSL_CERT_DIR pointing to nonexistent paths on some platforms; building a custom argo image that strips the trust store.

Related errors

• failed to append certificates from PEM string
• not implemented
• issuer empty
• clientID empty
• failed to create HTTP client: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/37432b24ec73809c. Report an issue: GitHub.