kubernetes/kops · error

waiting for kubernetes API to be served: %w

Error message

waiting for kubernetes API to be served: %w

What it means

After updating the control plane configuration, RunReconcileCluster validates that the Kubernetes API is being served (via RunValidateCluster with a 10-minute wait, only control-plane instance groups, pods ignored) before starting the control-plane rolling update. If that validation fails within the wait window, the underlying validation error is wrapped as "waiting for kubernetes API to be served: %w". This means the API server is not reachable/healthy after the update step, so reconcile cannot safely proceed to rolling updates.

Source

Thrown at cmd/kops/reconcile_cluster.go:172

	{
		opt := &ValidateClusterOptions{}
		opt.InitDefaults()
		opt.ClusterName = c.ClusterName
		opt.CreateKubecfgOptions = options.CreateKubecfgOptions
		opt.wait = 10 * time.Minute

		// filter the instance group to only include the control plane
		opt.filterInstanceGroups = func(ig *kops.InstanceGroup) bool {
			return ig.Spec.Role.HasAPIServer() || ig.Spec.Role.HasControlPlane()
		}

		// Ignore all pods, we just want to check the control plane is responding
		opt.filterPodsForValidation = func(pod *v1.Pod) bool {
			return false
		}

		if _, err := RunValidateCluster(ctx, f, out, opt); err != nil {
			return fmt.Errorf("waiting for kubernetes API to be served: %w", err)
		}
	}

	fmt.Fprintf(out, "Performing rolling-update for control plane\n")
	{
		opt := &RollingUpdateOptions{}
		opt.InitDefaults()
		opt.ClusterName = c.ClusterName
		opt.CreateKubecfgOptions = options.CreateKubecfgOptions
		opt.InstanceGroupRoles = []string{
			string(kops.InstanceGroupRoleAPIServer),
			string(kops.InstanceGroupRoleControlPlane),
		}
		opt.Yes = c.Yes
		if err := RunRollingUpdateCluster(ctx, f, out, opt); err != nil {
			return err
		}
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped error for the root cause; run `kops validate cluster <name>` manually to see current API health and which instance groups fail.
2. Check control-plane health: SSH/kubectl to the API server nodes and inspect kube-apiserver pods/logs (`kubectl -n kube-system get pods`, or via the node directly) for crash loops.
3. Verify network path to the API: DNS resolution of the cluster name, load balancer health, security groups/firewall allowing TCP 443, and valid kubeconfig credentials (`kops export kubecfg <name> --admin`).
4. If the API is merely slow to start, re-run `kops reconcile cluster --yes` once the API responds — validate-then-roll is idempotent and will resume at the same step.
5. For persistent failures, fix the underlying update issue (e.g. re-run update with corrections, check kOps version compatibility) before retrying reconcile.

Example fix

// before: reconcile fails while waiting for API
// kops reconcile cluster k8s.example.com --yes
// -> waiting for kubernetes API to be served: cluster "k8s.example.com" has no running instance groups...
// after: restore API access, then validate and retry
// kops export kubecfg k8s.example.com --admin
// kops validate cluster k8s.example.com
// kops reconcile cluster k8s.example.com --yes

// Before reconcile, confirm the API is already healthy:
// kops validate cluster <name>
// or programmatically: run RunValidateCluster and only proceed on success

if _, err := RunValidateCluster(ctx, f, out, opt); err != nil {
    // retry with backoff up to the wait budget before failing reconcile
    return retry.Do(
        func() error { return fmt.Errorf("waiting for kubernetes API to be served: %w", err) },
        retry.Attempts(5), retry.Delay(30*time.Second),
    )
}

When it happens

Trigger: RunValidateCluster returns an error during the post-update wait: API server pods crash-looping or not yet started after an update, unreachable API endpoint (DNS/LB not provisioned or misconfigured), kubeconfig/credentials cannot authenticate to the cluster, network/firewall blocking the API port, or validation timing out after 10 minutes on a slow control plane.

Common situations: Upgrading a cluster where the API server fails to come up (bad manifest, kube-apiserver image pull failure); new clusters where the load balancer or DNS record isn't ready yet; VPN/firewall rules preventing the local machine from reaching the API endpoint; expired or missing admin credentials in kubeconfig; heavily loaded or single-node control planes exceeding the 10-minute validation wait.

Related errors

• wait time exceeded during validation
• cluster passed validation %d consecutive times
• cluster not yet healthy
• cluster did not validate within a duration of %q
• error querying namespace %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/677cf49a5342837c. Report an issue: GitHub.