{"record":{"id":"6c13fb96fd0d8772","repo":"argoproj/argo-workflows","slug":"expected-1-pod-got-d-this-can-happen-when-multi","errorCode":null,"errorMessage":"expected 1 pod, got %d. This can happen when multiple workflow-controller pods are running and both reconciling this Workflow. Check your Argo Workflows installation for a rogue workflow-controller. Otherwise, this is a bug","messagePattern":"expected 1 pod, got (.+?)\\. This can happen when multiple workflow-controller pods are running and both reconciling this Workflow\\. Check your Argo Workflows installation for a rogue workflow-controller\\. Otherwise, this is a bug","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"workflow/controller/workflowpod.go","lineNumber":981,"sourceCode":"\t// active-pod accounting, progress write).\n\tresult.Pod = pod\n\treturn result, nil\n}\n\nfunc (woc *wfOperationCtx) podExists(nodeID string) (existing *apiv1.Pod, exists bool, err error) {\n\tobjs, err := woc.controller.PodController.GetPodsByIndex(indexes.NodeIDIndex, woc.wf.Namespace+\"/\"+nodeID)\n\tif err != nil {\n\t\treturn nil, false, fmt.Errorf(\"failed to get pod from informer store: %w\", err)\n\t}\n\n\tobjectCount := len(objs)\n\n\tif objectCount == 0 {\n\t\treturn nil, false, nil\n\t}\n\n\tif objectCount > 1 {\n\t\treturn nil, false, fmt.Errorf(\"expected 1 pod, got %d. This can happen when multiple workflow-controller \"+\n\t\t\t\"pods are running and both reconciling this Workflow. Check your Argo Workflows installation for a rogue \"+\n\t\t\t\"workflow-controller. Otherwise, this is a bug\", len(objs))\n\t}\n\n\tif existing, ok := objs[0].(*apiv1.Pod); ok {\n\t\treturn existing, true, nil\n\t}\n\n\treturn nil, false, nil\n}\n\nfunc (pb *podBuilder) getDeadline(opts *createWorkflowPodOpts) *time.Time {\n\tdeadline := time.Time{}\n\tif pb.in.workflowDeadline != nil && !opts.onExitPod {\n\t\tdeadline = *pb.in.workflowDeadline\n\t}\n\tif !opts.executionDeadline.IsZero() && (deadline.IsZero() || opts.executionDeadline.Before(deadline)) {\n\t\tdeadline = opts.executionDeadline","sourceCodeStart":963,"sourceCodeEnd":999,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/workflowpod.go#L963-L999","documentation":"podExists() found more than one pod in the informer index for a single node ID. Exactly one pod per workflow node is expected; multiple indicates either a rogue controller also creating pods or an internal bug, so the controller fails safe instead of guessing which pod is authoritative.","triggerScenarios":"GetPodsByIndex(NodeIDIndex, ns/nodeID) returns >1 objects while called from nodePodExist, executeTemplate, or createWorkflowPod — two controllers (possibly different versions/namespaces) both created a pod for the same node, or index pollution/duplicate keys.","commonSituations":"Accidentally running two workflow-controller deployments (e.g. during upgrade or in a second namespace watching the same cluster); leader-election misconfiguration; stale duplicate pods after crash-looping controller retries.","solutions":["Check for and remove rogue workflow-controller deployments: kubectl get deploy -A | grep workflow-controller","Verify leader election is enabled and only one controller holds the lease","Delete duplicate pods for the affected workflow and let the controller re-reconcile","Check pod names/creation timestamps on the workflow's nodes to identify which controller created the extras"],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"# detect rogue controllers\nkubectl get deploy -A -o name | grep workflow-controller\n# should be exactly one replica and one deployment","typeGuard":"// narrow the informer object and enforce count == 1\nif objectCount > 1 {\n    return nil, false, fmt.Errorf(\"expected 1 pod, got %d ...\", len(objs))\n}\npod, ok := objs[0].(*apiv1.Pod)\nif !ok {\n    return nil, false, fmt.Errorf(\"indexed object is not a Pod\")\n}","tryCatchPattern":"existing, exists, err := woc.podExists(nodeID)\nif err != nil {\n    // fail safe: investigate duplicates before proceeding\n    return err\n}","preventionTips":["Run exactly one workflow-controller deployment; use leader election","Never scale the controller above 1 replica without leader election","Clean up duplicate pods and re-reconcile affected workflows"],"tags":["kubernetes","controller-duplication","informer","consistency"],"backgroundTag":"rogue-controller-detected","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}