{"record":{"id":"de58163916534efd","repo":"tailscale/tailscale","slug":"failed-to-parse-pebble-minica-cert","errorCode":null,"errorMessage":"failed to parse pebble minica cert","messagePattern":"failed to parse pebble minica cert","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/k8s-operator/e2e/setup.go","lineNumber":213,"sourceCode":"\t)\n\ttestCAs = x509.NewCertPool()\n\tif *fDevcontrol {\n\t\t// Deploy pebble and get its certs.\n\t\tif err = applyPebbleResources(ctx, kubeClient); err != nil {\n\t\t\treturn 0, fmt.Errorf(\"failed to apply pebble resources: %w\", err)\n\t\t}\n\n\t\tpebblePod, err := waitForPodReady(ctx, logger, kubeClient, ns, client.MatchingLabels{\"app\": \"pebble\"})\n\t\tif err != nil {\n\t\t\treturn 0, fmt.Errorf(\"pebble pod not ready: %w\", err)\n\t\t}\n\n\t\tif err = forwardLocalPortToPod(ctx, logger, restCfg, ns, pebblePod, 15000); err != nil {\n\t\t\treturn 0, fmt.Errorf(\"failed to set up port forwarding to pebble: %w\", err)\n\t\t}\n\n\t\tif ok := testCAs.AppendCertsFromPEM(pebbleMiniCACert); !ok {\n\t\t\treturn 0, fmt.Errorf(\"failed to parse pebble minica cert\")\n\t\t}\n\n\t\tvar pebbleCAChain []byte\n\t\tfor _, path := range []string{\"/intermediates/0\", \"/roots/0\"} {\n\t\t\tpem, err := pebbleGet(ctx, 15000, path)\n\t\t\tif err != nil {\n\t\t\t\treturn 0, err\n\t\t\t}\n\t\t\tpebbleCAChain = append(pebbleCAChain, pem...)\n\t\t}\n\n\t\tif ok := testCAs.AppendCertsFromPEM(pebbleCAChain); !ok {\n\t\t\treturn 0, fmt.Errorf(\"failed to parse pebble ca chain cert\")\n\t\t}\n\n\t\tif err = os.MkdirAll(certsDir, 0755); err != nil {\n\t\t\treturn 0, fmt.Errorf(\"failed to create certs dir: %w\", err)\n\t\t}","sourceCodeStart":195,"sourceCodeEnd":231,"githubUrl":"https://github.com/tailscale/tailscale/blob/cfe32b8be6a33f8e24fbc369cbfbf7c729d9e042/cmd/k8s-operator/e2e/setup.go#L195-L231","documentation":"E2E-setup error returned when x509.CertPool.AppendCertsFromPEM(pebbleMiniCACert) returns false — the embedded pebbleMiniCACert constant could not be parsed as any PEM certificate. Since it is a compile-time constant in the e2e package, failure implies the constant itself is malformed rather than anything environment-dependent.","triggerScenarios":"AppendCertsFromPEM parses only CERTIFICATE PEM blocks; it returns false when the bytes are empty, truncated, have a bad PEM frame, or contain only other block types. With the shipped constant this should not occur; forks that regenerate or hand-edit pebbleMiniCACert (e.g. switching pebble versions with different test CA keys) can break it.","commonSituations":"Forking the e2e suite and pasting a new CA cert without the full BEGIN/END CERTIFICATE framing; accidental whitespace/escaping damage in the raw-string literal; build tooling mangling the source file.","solutions":["Regenerate the constant from a valid PEM: openssl x509 -in mini-ca.pem -outform PEM, then embed the complete BEGIN..END block.","Sanity-check in Go: pem.Decode on the constant must return a *pem.Block of type CERTIFICATE.","If untouched upstream code hits this, report a bug with your Go version and any source patching (go:generate, lint --fix)."],"exampleFix":"// before: broken literal (truncated or missing PEM frame)\nconst pebbleMiniCACert = \"-----BEGIN CERTIFICATE-----MIIB...\" // no END line\n\n// after: complete PEM block\nconst pebbleMiniCACert = `-----BEGIN CERTIFICATE-----\nMIIB...\n-----END CERTIFICATE-----\n`","handlingStrategy":"validation","validationCode":"// Guard the constant once in a unit test so breakage is caught at build time\nfunc TestPebbleMiniCACertIsValid(t *testing.T) {\n\tok := x509.NewCertPool().AppendCertsFromPEM(pebbleMiniCACert)\n\tif !ok {\n\t\tt.Fatal(\"pebbleMiniCACert is not parseable PEM\")\n\t}\n}","typeGuard":"func isValidPEMCert(pemBytes []byte) bool {\n\tblock, _ := pem.Decode(pemBytes)\n\tif block == nil || block.Type != \"CERTIFICATE\" {\n\t\treturn false\n\t}\n\t_, err := x509.ParseCertificate(block.Bytes)\n\treturn err == nil\n}","tryCatchPattern":"if ok := testCAs.AppendCertsFromPEM(pebbleMiniCACert); !ok {\n\t// constant is compile-time data: fail fast with a clear message\n\treturn 0, fmt.Errorf(\"pebbleMiniCACert embedded in the e2e package is invalid; regenerate it from the pebble mini CA PEM\")\n}","preventionTips":["Add a unit test asserting the embedded CA parses (pem.Decode + x509.ParseCertificate).","When forking, embed the complete BEGIN/END CERTIFICATE block via go:embed of a .pem file instead of raw strings.","Verify embedded certs after any pebble version bump or key rotation."],"tags":["e2e","testing","tls","pebble","pem"],"backgroundTag":null,"analyzedSha":"cfe32b8be6a33f8e24fbc369cbfbf7c729d9e042","analyzedAt":"2026-08-15T19:58:31.583Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}