{"record":{"id":"37432b24ec73809c","repo":"argoproj/argo-workflows","slug":"failed-to-load-system-cert-pool-w","errorCode":null,"errorMessage":"failed to load system cert pool: %w","messagePattern":"failed to load system cert pool: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/auth/sso/clients.go","lineNumber":41,"sourceCode":"\t\t}\n\t}\n\n\treturn fmt.Sprintf(\"HTTPClientConfig{InsecureSkipVerify: %t, RootCA: %q (%d bytes)}\",\n\t\tc.InsecureSkipVerify, rootCAPreview, rootCALen)\n}\n\nfunc createHTTPClient(config HTTPClientConfig) (*http.Client, error) {\n\t// Start with a copy of the default client\n\thttpClient := *http.DefaultClient\n\n\t// Clone the default transport and cast to *http.Transport\n\tdefaultTransport := http.DefaultTransport.(*http.Transport)\n\ttransport := defaultTransport.Clone()\n\n\t// Load system cert pool to respect env.SSL_CERT_DIR, env.SSL_CERT_FILE. macOS are not supported (https://pkg.go.dev/crypto/x509#SystemCertPool)\n\trootCAs, err := x509.SystemCertPool()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to load system cert pool: %w\", err)\n\t}\n\n\t// Set RootCAs if provided\n\t// Load root CA certificates from PEM string if defined\n\tif config.RootCA != \"\" {\n\t\tif ok := rootCAs.AppendCertsFromPEM([]byte(config.RootCA)); !ok {\n\t\t\treturn nil, fmt.Errorf(\"failed to append certificates from PEM string\")\n\t\t}\n\t}\n\n\t// Apply the custom TLS config to the cloned transport\n\ttransport.TLSClientConfig = &tls.Config{\n\t\tInsecureSkipVerify: config.InsecureSkipVerify,\n\t\tRootCAs:            rootCAs,\n\t}\n\n\t// Use the modified transport in our client copy\n\thttpClient.Transport = transport","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/server/auth/sso/clients.go#L23-L59","documentation":"The SSO HTTP client builder clones http.DefaultTransport and loads the OS root certificate pool via x509.SystemCertPool() so that SSL_CERT_DIR/SSL_CERT_FILE are respected. If the system pool cannot be loaded (e.g. no root store available on the platform), the error is wrapped as 'failed to load system cert pool: %w' and SSO client creation fails. Per Go docs, SystemCertPool is unsupported on macOS and can fail on systems without any CA bundle.","triggerScenarios":"createHTTPClient (during SSO init at argo-server startup or in tests) calls x509.SystemCertPool() on a platform/manifest combination with no CA bundle: minimal containers lacking /etc/ssl/certs, musl images without ca-certificates installed, or macOS builds where it returns an error.","commonSituations":"Running argo-server in a distroless/slim image without the ca-certificates package; SSL_CERT_FILE/SSL_CERT_DIR pointing to nonexistent paths on some platforms; building a custom argo image that strips the trust store.","solutions":["Install CA certificates in the container image: `apt-get install -y ca-certificates` (Debian) or the equivalent for your base image","Set SSL_CERT_FILE to a valid PEM bundle (e.g. the image's ca-certificates.crt) and ensure the file exists","Provide config.RootCA in the SSO ConfigMap and, if the platform still fails, add explicit fallback handling for SystemCertPool errors"],"exampleFix":"// Dockerfile\n// before\nFROM debian:slim\n// after\nFROM debian:slim\nRUN apt-get update && apt-get install -y ca-certificates","handlingStrategy":"validation","validationCode":"if _, err := os.Stat(os.Getenv(\"SSL_CERT_FILE\")); os.Getenv(\"SSL_CERT_FILE\") != \"\" && err != nil {\n    log.Printf(\"SSL_CERT_FILE points to missing file: %v\", err)\n}\nif _, err := x509.SystemCertPool(); err != nil {\n    log.Printf(\"system cert pool unavailable: %v\", err)\n}","typeGuard":"func systemTrustAvailable() bool {\n    _, err := x509.SystemCertPool()\n    return err == nil\n}","tryCatchPattern":"client, err := createHTTPClient(cfg)\nif err != nil && strings.Contains(err.Error(), \"failed to load system cert pool\") {\n    // install ca-certificates in the image or set SSL_CERT_FILE to a valid bundle\n}","preventionTips":["Always install ca-certificates in container images that create TLS clients","Don't run Go SystemCertPool-dependent code on macOS targets (per pkg.go.dev docs)","Set SSL_CERT_FILE/SSL_CERT_DIR only to files that exist in the image","Smoke-test SSO startup in CI using the same minimal base image"],"tags":["tls","certificates","sso","oidc","container"],"backgroundTag":"system-cert-pool-load-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}