kubernetes/kops · error

failed to read %q certificates: %w

Error message

failed to read %q certificates: %w

What it means

After finding the requested keyset, loadCertificates calls keyset.ToCertificateBytes() to serialize the CA certificates into PEM bytes for config.CAs. This error wraps any failure of that serialization — typically a keyset with no usable certificate data or corrupted items.

Source

Thrown at pkg/nodemodel/nodeupconfigbuilder.go:470

		baseURL := url.URL{
			Scheme: "https",
			Host:   net.JoinHostPort(host, strconv.Itoa(wellknownports.KopsControllerPort)),
			Path:   "/",
		}
		configServer.Servers = append(configServer.Servers, baseURL.String())
	}

	return configServer
}

func loadCertificates(keysets map[string]*fi.Keyset, name string, config *nodeup.Config, includeKeypairID bool) error {
	keyset := keysets[name]
	if keyset == nil {
		return fmt.Errorf("key %q not found", name)
	}
	certificates, err := keyset.ToCertificateBytes()
	if err != nil {
		return fmt.Errorf("failed to read %q certificates: %w", name, err)
	}
	config.CAs[name] = string(certificates)
	if includeKeypairID {
		if keyset.Primary == nil || keyset.Primary.Id == "" {
			return fmt.Errorf("key %q did not have primary id set", name)
		}
		config.KeypairIDs[name] = keyset.Primary.Id
	}
	return nil
}

// buildWarmPoolImages returns a list of container images that should be pre-pulled during instance pre-initialization
func (n *nodeUpConfigBuilder) buildWarmPoolImages(ig *kops.InstanceGroup) []string {
	if ig == nil || ig.Spec.Role.HasControlPlane() {
		return nil
	}

	images := map[string]bool{}

View on GitHub (pinned to 4c8573c808)

Solutions

  1. Inspect the wrapped error in the message to distinguish storage I/O issues from malformed data; fix state-store access (credentials, permissions) if it is an I/O error.
  2. Regenerate the broken keyset: delete the corrupt keypair (`kops delete secret <name>` / keypair API) then `kops update cluster --yes` to recreate it.
  3. Restore the keyset object from the state-store backup or versioning (S3 object versions) to a state with valid certificate items.
  4. Run `kops toolbox dump` / inspect the keyset to confirm the primary item has non-empty cert data before retrying.

Example fix

// before: corrupt keyset in state store
s3://bucket/cluster/keys/kubernetes-ca/  (empty item.yaml)
// after
$ kops delete secret kubernetes-ca --yes && kops update cluster --yes  # recreated with valid PEM
Defensive patterns

Strategy: try-catch

Validate before calling

ks := keysets[name]
if ks == nil {
    return fmt.Errorf("keyset %q missing", name)
}
if ks.Primary == nil || len(ks.Primary.Certificates) == 0 {
    return fmt.Errorf("keyset %q has no certificate data; recreate it with kops update cluster", name)
}

Type guard

func keysetHasCertificates(ks *fi.Keyset) bool {
    return ks != nil && ks.Primary != nil && len(ks.Primary.Certificates) > 0
}

Try / catch

if err := loadCertificates(keysets, name, config, true); err != nil {
    if strings.Contains(err.Error(), "failed to read") {
        // keyset exists but data unreadable/corrupt: check state-store health, recreate keyset
        return fmt.Errorf("keyset %q unreadable; verify state store or recreate keypair: %w", name, err)
    }
    return err
}

Prevention

When it happens

Trigger: BuildConfig -> loadCertificates on a keyset that exists in the map but whose items cannot be converted to certificate bytes: keyset with zero items, items with malformed/empty PEM data, or an underlying storage read failure while materializing the keyset items.

Common situations: Interrupted `kops update cluster` left a keyset created but never populated with certificate data; state-store object corruption or truncated writes (S3/GCS/etcd backend); manually editing files under the state store's keys/ directory; decrypt/VFS read errors surfaced through the wrapped error.

Understand the failure class

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/97a3735de072551c. Report an issue: GitHub.