{"record":{"id":"a1b8af44f2e8c21f","repo":"argoproj/argo-workflows","slug":"failed-to-append-certificates-from-pem-string","errorCode":null,"errorMessage":"failed to append certificates from PEM string","messagePattern":"failed to append certificates from PEM string","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/auth/sso/clients.go","lineNumber":48,"sourceCode":"func createHTTPClient(config HTTPClientConfig) (*http.Client, error) {\n\t// Start with a copy of the default client\n\thttpClient := *http.DefaultClient\n\n\t// Clone the default transport and cast to *http.Transport\n\tdefaultTransport := http.DefaultTransport.(*http.Transport)\n\ttransport := defaultTransport.Clone()\n\n\t// Load system cert pool to respect env.SSL_CERT_DIR, env.SSL_CERT_FILE. macOS are not supported (https://pkg.go.dev/crypto/x509#SystemCertPool)\n\trootCAs, err := x509.SystemCertPool()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to load system cert pool: %w\", err)\n\t}\n\n\t// Set RootCAs if provided\n\t// Load root CA certificates from PEM string if defined\n\tif config.RootCA != \"\" {\n\t\tif ok := rootCAs.AppendCertsFromPEM([]byte(config.RootCA)); !ok {\n\t\t\treturn nil, fmt.Errorf(\"failed to append certificates from PEM string\")\n\t\t}\n\t}\n\n\t// Apply the custom TLS config to the cloned transport\n\ttransport.TLSClientConfig = &tls.Config{\n\t\tInsecureSkipVerify: config.InsecureSkipVerify,\n\t\tRootCAs:            rootCAs,\n\t}\n\n\t// Use the modified transport in our client copy\n\thttpClient.Transport = transport\n\n\treturn &httpClient, nil\n}\n","sourceCodeStart":30,"sourceCodeEnd":63,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/server/auth/sso/clients.go#L30-L63","documentation":"When the SSO config defines RootCA as a PEM string, createHTTPClient calls rootCAs.AppendCertsFromPEM, which returns false if no certificates could be decoded from the string. The function then returns 'failed to append certificates from PEM string'. AppendCertsFromPEM silently skips invalid entries, so this error means the entire RootCA value contained zero valid PEM certificates.","triggerScenarios":"sso.rootCA in the workflow-controller/argo-server ConfigMap is set but its value is not one or more '-----BEGIN CERTIFICATE-----' PEM blocks — e.g. it holds a private key, a raw base64 body without headers, an empty string with whitespace, or a URL/file path instead of the certificate contents.","commonSituations":"Users paste the CA file path rather than its contents into rootCA; the ConfigMap value was base64-encoded one time too many; line-wrapping corrupted the PEM; self-signed CA exported as PKCS#7/DER instead of PEM.","solutions":["Set rootCA to the literal PEM text: `cat ca.crt` output including BEGIN/END CERTIFICATE lines, in the argo-server SSO ConfigMap","If your value is base64, decode it once before storing: `base64 -d ca.b64 > ca.crt`, then paste ca.crt contents","Validate with `openssl x509 -in ca.crt -noout` and ensure the value is a certificate, not a key or CSR"],"exampleFix":"# ConfigMap\n# before\nrootCA: /etc/certs/ca.crt          # a path — wrong\n# after\nrootCA: |\n  -----BEGIN CERTIFICATE-----\n  ...\n  -----END CERTIFICATE-----","handlingStrategy":"validation","validationCode":"if cfg.RootCA != \"\" && !x509.NewCertPool().AppendCertsFromPEM([]byte(cfg.RootCA)) {\n    return errors.New(\"sso.rootCA does not contain any valid PEM certificates\")\n}","typeGuard":"func isValidRootCAPEM(pemStr string) bool {\n    pool := x509.NewCertPool()\n    return pemStr == \"\" || pool.AppendCertsFromPEM([]byte(pemStr))\n}","tryCatchPattern":"client, err := createHTTPClient(cfg)\nif err != nil && strings.Contains(err.Error(), \"failed to append certificates\") {\n    // log the HTTPClientConfig.String() preview and fix rootCA in the ConfigMap\n}","preventionTips":["Paste the literal PEM contents (with BEGIN/END lines) into rootCA, not a file path","Decode base64 exactly once before storing a CA in a ConfigMap","Validate the CA with `openssl x509 -in ca.crt -noout` before configuring","Use `kubectl create configmap --from-file=rootCA=ca.crt` so the PEM format is preserved"],"tags":["tls","sso","oidc","pem","configuration"],"backgroundTag":"invalid-root-ca-pem","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}