deployment %s is updated but rollout has not started

Error message

deployment %s is updated but rollout has not started

What it means

cilium-cli compares the Deployment's metadata.Generation with status.ObservedGeneration. When they differ, the Kubernetes DeploymentController has not yet processed the latest spec change, so the collector raises this aggregated error. It is a transient controller-lag signal rather than a pod health problem.

Source

Thrown at cilium-cli/status/k8s.go:184

	stateCount.Unavailable = int(d.Status.UnavailableReplicas)

	status.mutex.Lock()
	defer status.mutex.Unlock()

	status.PodState[name] = stateCount

	notReady := stateCount.Desired - stateCount.Ready
	if notReady > 0 {
		status.AddAggregatedError(name, name, fmt.Errorf("%d pods of Deployment %s are not ready", notReady, name))
	}

	if unavailable := stateCount.Unavailable - notReady; unavailable > 0 {
		status.AddAggregatedWarning(name, name, fmt.Errorf("%d pods of Deployment %s are not available", unavailable, name))
	}

	// ObservedGeneration behind: DeploymentController has not yet noticed the latest change
	if d.Generation != d.Status.ObservedGeneration {
		status.AddAggregatedError(name, name, fmt.Errorf("deployment %s is updated but rollout has not started", name))
	}

	// Deployment change is not fully rolled out
	if d.Status.UpdatedReplicas < d.Status.Replicas {
		status.AddAggregatedError(name, name, fmt.Errorf("deployment %s is rolling out - %d out of %d pods updated", name, d.Status.UpdatedReplicas, d.Status.Replicas))
	}

	return false, nil
}

func (k *K8sStatusCollector) podCount(ctx context.Context, status *Status) error {
	var numberAllPod, numberCiliumPod int

	pods, err := k.client.ListPods(ctx, "", metav1.ListOptions{})
	if err != nil {
		return err
	}

View on GitHub (pinned to ac7b90affa)

Solutions

Wait a few seconds and re-run `cilium status`; the controller usually catches up quickly.
Check the controller is healthy: `kubectl get pods -n kube-system | grep kube-controller-manager` and its logs.
Verify the Deployment's annotations/status with `kubectl get deployment <name> -o yaml` to see ObservedGeneration vs metadata.generation.
If it persists across minutes, investigate kube-controller-manager connectivity to the API server.

Example fix

// before
cilium upgrade && cilium status   # races the controller
// after
cilium upgrade
kubectl rollout status deployment/cilium -n kube-system
cilium status

Defensive patterns

Strategy: retry

Validate before calling

d, _ := clientset.AppsV1().Deployments(ns).Get(ctx, "cilium", metav1.GetOptions{})
if d != nil && d.Generation != d.Status.ObservedGeneration {
    // controller lag; retry shortly
}

Try / catch

if err := status.Check(ctx); err != nil && strings.Contains(err.Error(), "rollout has not started") {
    time.Sleep(10 * time.Second) // let DeploymentController observe the change
    err = status.Check(ctx)
}

Prevention

Insert a delay between install/upgrade and status checks.
Ensure kube-controller-manager is healthy in the cluster.
Don't interpret a single early status run as a failed install.

When it happens

Trigger: deploymentStatus read a Deployment immediately after its spec changed (d.Generation != d.Status.ObservedGeneration), e.g. `cilium upgrade`/config change moments before `cilium status` ran.

Common situations: Running `cilium status` immediately after `cilium install`/`upgrade`; a slow or wedged kube-controller-manager; API server/controller backpressure on busy clusters.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/43cddf02e1acf864. Report an issue: GitHub.