What causes "deployment %s is rolling out - %d out of %d pods updated" in cilium/cilium?

Track rollout progress: `kubectl rollout status deployment/ -n ` and wait for completion.

deployment %s is rolling out - %d out of %d pods updated

Error message

deployment %s is rolling out - %d out of %d pods updated

What it means

cilium-cli compares status.UpdatedReplicas with status.Replicas for each Deployment. When fewer pods than the total desired count run the new template, the deployment is mid-rollout and this aggregated error is recorded. It indicates a Cilium upgrade/config change is still propagating through the cluster.

Source

Thrown at cilium-cli/status/k8s.go:189

	status.PodState[name] = stateCount

	notReady := stateCount.Desired - stateCount.Ready
	if notReady > 0 {
		status.AddAggregatedError(name, name, fmt.Errorf("%d pods of Deployment %s are not ready", notReady, name))
	}

	if unavailable := stateCount.Unavailable - notReady; unavailable > 0 {
		status.AddAggregatedWarning(name, name, fmt.Errorf("%d pods of Deployment %s are not available", unavailable, name))
	}

	// ObservedGeneration behind: DeploymentController has not yet noticed the latest change
	if d.Generation != d.Status.ObservedGeneration {
		status.AddAggregatedError(name, name, fmt.Errorf("deployment %s is updated but rollout has not started", name))
	}

	// Deployment change is not fully rolled out
	if d.Status.UpdatedReplicas < d.Status.Replicas {
		status.AddAggregatedError(name, name, fmt.Errorf("deployment %s is rolling out - %d out of %d pods updated", name, d.Status.UpdatedReplicas, d.Status.Replicas))
	}

	return false, nil
}

func (k *K8sStatusCollector) podCount(ctx context.Context, status *Status) error {
	var numberAllPod, numberCiliumPod int

	pods, err := k.client.ListPods(ctx, "", metav1.ListOptions{})
	if err != nil {
		return err
	}

	if pods != nil && len(pods.Items) != 0 {
		for _, pod := range pods.Items {
			if !pod.Spec.HostNetwork && (pod.Status.Phase == corev1.PodRunning || pod.Status.Phase == corev1.PodPending) {
				numberAllPod++
			}

View on GitHub (pinned to ac7b90affa)

Solutions

Track rollout progress: `kubectl rollout status deployment/<name> -n <ns>` and wait for completion.
If stuck, inspect the new pods' events/logs (`kubectl describe pod`, `kubectl logs`) for image pull, scheduling, or crash-loop causes.
Roll back if the new version is broken: `kubectl rollout undo deployment/<name> -n <ns>`.
Re-run `cilium status` after the rollout completes.

Example fix

// before: assuming failure mid-upgrade
cilium status   # reports rolling out
// after
kubectl rollout status deployment/cilium -n kube-system
cilium status

Defensive patterns

Strategy: retry

Validate before calling

d, _ := clientset.AppsV1().Deployments(ns).Get(ctx, "cilium", metav1.GetOptions{})
if d != nil && d.Status.UpdatedReplicas < d.Status.Replicas {
    // rollout still in progress; wait
}

Try / catch

if err := status.Check(ctx); err != nil && strings.Contains(err.Error(), "is rolling out") {
    _ = wait.PollUntilContextTimeout(ctx, 10*time.Second, 10*time.Minute, true,
        func(ctx context.Context) (bool, error) { return rolloutComplete(ctx), nil })
    err = status.Check(ctx)
}

Prevention

Wait for `kubectl rollout status` completion after upgrades before status checks.
Schedule upgrades so rollouts finish before dependent automation runs.
Investigate stuck pods if the rollout doesn't converge in minutes.

When it happens

Trigger: deploymentStatus found d.Status.UpdatedReplicas < d.Status.Replicas for the named Deployment, e.g. 1 of 3 pods updated after a `cilium upgrade`.

Common situations: Checking status during an in-progress `cilium upgrade` or `cilium config` change; a stuck rollout where one or more pods cannot schedule or pass readiness on the new version.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/ee5ef0fc26ddfe1c. Report an issue: GitHub.