kubernetes/kops · critical

reading %q certificate: %v

Error message

reading %q certificate: %v

What it means

During startup, newKeystore reads each configured CA's certificate file (<basePath>/<name>.crt) from disk. If that read fails — file missing, wrong permissions, or wrong path — startup aborts with "reading %q certificate: %v" wrapping the underlying OS error.

Source

Thrown at cmd/kops-controller/pkg/server/keystore.go:86

// MirrorTo will copy secrets to a vfs.Path, which is often easier for a machine to read
func (k *keystore) MirrorTo(ctx context.Context, basedir vfs.Path) error {
	return fmt.Errorf("server-side client does not support MirrorTo")
}

// ListKeysets will return all the KeySets.
func (k *keystore) ListKeysets() (map[string]*fi.Keyset, error) {
	return nil, fmt.Errorf("server-side client does not support ListKeysets")
}

func newKeystore(basePath string, cas []string) (*keystore, map[string]string, error) {
	keystore := &keystore{
		keys:    map[string]keystoreEntry{},
		keySets: map[string]*fi.Keyset{},
	}
	for _, name := range cas {
		certBytes, err := os.ReadFile(path.Join(basePath, name+".crt"))
		if err != nil {
			return nil, nil, fmt.Errorf("reading %q certificate: %v", name, err)
		}
		// TODO: Support multiple certificates?
		certificate, err := pki.ParsePEMCertificate(certBytes)
		if err != nil {
			return nil, nil, fmt.Errorf("parsing %q certificate: %v", name, err)
		}

		keyBytes, err := os.ReadFile(path.Join(basePath, name+".key"))
		if err != nil {
			return nil, nil, fmt.Errorf("reading %q key: %v", name, err)
		}
		key, err := pki.ParsePEMPrivateKey(keyBytes)
		if err != nil {
			return nil, nil, fmt.Errorf("parsing %q key: %v", name, err)
		}

		keystore.keys[name] = keystoreEntry{
			certificate: certificate,

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped OS error in the message: if it says "no such file or directory", confirm <basePath>/<name>.crt exists with exactly that name
2. Verify the controller's basePath flag/mount points at the directory containing the CA secrets (e.g. /etc/kubernetes/pki or the mounted secret volume)
3. Check file permissions/ownership so the controller user can read the .crt files
4. Re-provision or copy the missing CA certificate file and restart kops-controller

Example fix

// before: controller config
--pki-dir=/etc/kubernetes/pki  # but ca.crt placed as /etc/kubernetes/pki/kubernetes-ca.cert

// after
mv /etc/kubernetes/pki/kubernetes-ca.cert /etc/kubernetes/pki/kubernetes-ca.crt
# ensure file exists: ls /etc/kubernetes/pki/kubernetes-ca.crt

// Pre-flight before starting the server
for _, name := range cas {
	crt := filepath.Join(basePath, name+".crt")
	key := filepath.Join(basePath, name+".key")
	for _, f := range []string{crt, key} {
		if fi, err := os.Stat(f); err != nil {
			return fmt.Errorf("PKI preflight failed: %s: %w", f, err)
		} else if fi.IsDir() {
			return fmt.Errorf("PKI preflight failed: %s is a directory", f)
		}
	}
}

func caFilesReadable(basePath, name string) bool {
	for _, suffix := range []string{".crt", ".key"} {
		f, err := os.Open(path.Join(basePath, name+suffix))
		if err != nil {
			return false
		}
		f.Close()
	}
	return true
}

keystore, keypairIDs, err := newKeystore(basePath, cas)
if err != nil {
	if strings.Contains(err.Error(), "certificate: ") && errors.Is(unwrap(err), os.ErrNotExist) {
		klog.Fatalf("CA certificate missing in %s: %v", basePath, err)
	}
	return err
}

When it happens

Trigger: newKeystore(basePath, cas) is invoked by NewServer and os.ReadFile fails for a configured CA: the .crt file does not exist in basePath, the name does not match the filename, basePath itself is wrong, or the controller process lacks read permission on the file.

Common situations: Controller started with a wrong PKI directory (e.g. wrong mount in a container image); CA file named differently than the configured CA name (case/extension mismatch); secrets volume not mounted; permissions too restrictive after secret provisioning.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• error building metal node identifier: %w
• error building linode node identifier: %w
• must specify cloud
• identifier for cloud %q not implemented
• unknown CA %q

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/f6f7d06d47d30a4b. Report an issue: GitHub.