{"record":{"id":"f195d9d9d14558bb","repo":"argoproj/argo-workflows","slug":"agent-pod-failed-with-reason-s","errorCode":null,"errorMessage":"agent pod failed with reason:\"%s\"","messagePattern":"agent pod failed with reason:\"(.+?)\"","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/controller/agent.go","lineNumber":54,"sourceCode":"\tif len(woc.taskSet) == 0 {\n\t\treturn nil\n\t}\n\tpod, err := woc.createAgentPod(ctx)\n\tif err != nil {\n\t\treturn err\n\t}\n\t// Check Pod is just created\n\tif pod != nil && pod.Status.Phase != \"\" {\n\t\twoc.updateAgentPodStatus(ctx, pod)\n\t}\n\treturn nil\n}\n\nfunc (woc *wfOperationCtx) updateAgentPodStatus(ctx context.Context, pod *apiv1.Pod) {\n\twoc.log.Info(ctx, \"updateAgentPodStatus\")\n\tnewPhase, message := assessAgentPodStatus(ctx, pod)\n\tif newPhase == wfv1.NodeFailed || newPhase == wfv1.NodeError {\n\t\twoc.markTaskSetNodesError(ctx, fmt.Errorf(`agent pod failed with reason:\"%s\"`, message))\n\t}\n}\n\nfunc assessAgentPodStatus(ctx context.Context, pod *apiv1.Pod) (wfv1.NodePhase, string) {\n\tvar newPhase wfv1.NodePhase\n\tvar message string\n\tlogger := logging.RequireLoggerFromContext(ctx)\n\tlogger.WithField(\"namespace\", pod.Namespace).\n\t\tWithField(\"podName\", pod.Name).\n\t\tInfo(ctx, \"assessAgentPodStatus\")\n\tswitch pod.Status.Phase {\n\tcase apiv1.PodSucceeded, apiv1.PodRunning, apiv1.PodPending:\n\t\treturn \"\", \"\"\n\tcase apiv1.PodFailed:\n\t\tnewPhase = wfv1.NodeFailed\n\t\tmessage = pod.Status.Message\n\tdefault:\n\t\tnewPhase = wfv1.NodeError","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/agent.go#L36-L72","documentation":"The controller runs a dedicated 'Agent' pod to execute HTTP/plugin template actions. updateAgentPodStatus watches that pod; when assessAgentPodStatus concludes the agent pod has Failed or Errored, the controller fails all pending TaskSet nodes with 'agent pod failed with reason:\"%s\"'. This error is the propagated reason string from the agent pod's own status (e.g. its container terminated with a message).","triggerScenarios":"The agent pod's container exits non-zero or is evicted/OOMKilled while HTTP template or executor-plugin tasks are in flight; reconcileAgentPod reads the updated pod and calls updateAgentPodStatus, which marks all WorkflowTaskSet nodes errored with the pod's reason message.","commonSituations":"Agent pod OOM-killed due to many concurrent HTTP actions; node pressure or eviction; the agent image failing to start (image pull error, RBAC missing for plugin connections); cluster autoscaler removing the agent node mid-run.","solutions":["Inspect the agent pod (`kubectl describe pod <agent-pod> -n <wf-namespace>`) to find the underlying reason (OOMKilled, Evicted, image error) and fix that cause","Retry the workflow — agent pod failures are usually transient infra issues, not workflow definition problems","If OOM, increase controller resource requests/limits for agent pods or reduce concurrency of HTTP/plugin tasks","If image pull issues, verify the argoexec image is available in the cluster/registry"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// controller-side: nodes are already marked errored; on the client side handle with retry\ntry {\n    await wf.wait()\n} catch (e) {\n    if (/agent pod failed/.test(e.message)) retryWorkflow(wfName)\n}","preventionTips":["Set adequate resources for agent pods and avoid oversubscribed nodes","Monitor pod evictions/OOMKills in the workflow namespace","Reduce concurrent HTTP/plugin task fan-out","Use retryStrategy on workflows susceptible to infra failures"],"tags":["kubernetes","pod-failure","agent","argo-workflows"],"backgroundTag":"pod-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}