argoproj/argo-workflows · critical

expected 1 pod, got %d. This can happen when multiple workfl

Error message

expected 1 pod, got %d. This can happen when multiple workflow-controller pods are running and both reconciling this Workflow. Check your Argo Workflows installation for a rogue workflow-controller. Otherwise, this is a bug

What it means

podExists() found more than one pod in the informer index for a single node ID. Exactly one pod per workflow node is expected; multiple indicates either a rogue controller also creating pods or an internal bug, so the controller fails safe instead of guessing which pod is authoritative.

Source

Thrown at workflow/controller/workflowpod.go:981

	// active-pod accounting, progress write).
	result.Pod = pod
	return result, nil
}

func (woc *wfOperationCtx) podExists(nodeID string) (existing *apiv1.Pod, exists bool, err error) {
	objs, err := woc.controller.PodController.GetPodsByIndex(indexes.NodeIDIndex, woc.wf.Namespace+"/"+nodeID)
	if err != nil {
		return nil, false, fmt.Errorf("failed to get pod from informer store: %w", err)
	}

	objectCount := len(objs)

	if objectCount == 0 {
		return nil, false, nil
	}

	if objectCount > 1 {
		return nil, false, fmt.Errorf("expected 1 pod, got %d. This can happen when multiple workflow-controller "+
			"pods are running and both reconciling this Workflow. Check your Argo Workflows installation for a rogue "+
			"workflow-controller. Otherwise, this is a bug", len(objs))
	}

	if existing, ok := objs[0].(*apiv1.Pod); ok {
		return existing, true, nil
	}

	return nil, false, nil
}

func (pb *podBuilder) getDeadline(opts *createWorkflowPodOpts) *time.Time {
	deadline := time.Time{}
	if pb.in.workflowDeadline != nil && !opts.onExitPod {
		deadline = *pb.in.workflowDeadline
	}
	if !opts.executionDeadline.IsZero() && (deadline.IsZero() || opts.executionDeadline.Before(deadline)) {
		deadline = opts.executionDeadline

View on GitHub (pinned to 35bff19146)

Solutions

1. Check for and remove rogue workflow-controller deployments: kubectl get deploy -A | grep workflow-controller
2. Verify leader election is enabled and only one controller holds the lease
3. Delete duplicate pods for the affected workflow and let the controller re-reconcile
4. Check pod names/creation timestamps on the workflow's nodes to identify which controller created the extras

# detect rogue controllers
kubectl get deploy -A -o name | grep workflow-controller
# should be exactly one replica and one deployment

// narrow the informer object and enforce count == 1
if objectCount > 1 {
    return nil, false, fmt.Errorf("expected 1 pod, got %d ...", len(objs))
}
pod, ok := objs[0].(*apiv1.Pod)
if !ok {
    return nil, false, fmt.Errorf("indexed object is not a Pod")
}

existing, exists, err := woc.podExists(nodeID)
if err != nil {
    // fail safe: investigate duplicates before proceeding
    return err
}

When it happens

Trigger: GetPodsByIndex(NodeIDIndex, ns/nodeID) returns >1 objects while called from nodePodExist, executeTemplate, or createWorkflowPod — two controllers (possibly different versions/namespaces) both created a pod for the same node, or index pollution/duplicate keys.

Common situations: Accidentally running two workflow-controller deployments (e.g. during upgrade or in a second namespace watching the same cluster); leader-election misconfiguration; stale duplicate pods after crash-looping controller retries.

Related errors

• failed to get pod from informer store: %w
• object %+v is not an unstructured
• failed to get workflow by key after locking
• expected no more than 1 workflow, got %d
• failed to get pod from informer store: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/6c13fb96fd0d8772. Report an issue: GitHub.