argoproj/argo-workflows · error

agent pod failed with reason:"%s"

Error message

agent pod failed with reason:"%s"

What it means

The controller runs a dedicated 'Agent' pod to execute HTTP/plugin template actions. updateAgentPodStatus watches that pod; when assessAgentPodStatus concludes the agent pod has Failed or Errored, the controller fails all pending TaskSet nodes with 'agent pod failed with reason:"%s"'. This error is the propagated reason string from the agent pod's own status (e.g. its container terminated with a message).

Source

Thrown at workflow/controller/agent.go:54

	if len(woc.taskSet) == 0 {
		return nil
	}
	pod, err := woc.createAgentPod(ctx)
	if err != nil {
		return err
	}
	// Check Pod is just created
	if pod != nil && pod.Status.Phase != "" {
		woc.updateAgentPodStatus(ctx, pod)
	}
	return nil
}

func (woc *wfOperationCtx) updateAgentPodStatus(ctx context.Context, pod *apiv1.Pod) {
	woc.log.Info(ctx, "updateAgentPodStatus")
	newPhase, message := assessAgentPodStatus(ctx, pod)
	if newPhase == wfv1.NodeFailed || newPhase == wfv1.NodeError {
		woc.markTaskSetNodesError(ctx, fmt.Errorf(`agent pod failed with reason:"%s"`, message))
	}
}

func assessAgentPodStatus(ctx context.Context, pod *apiv1.Pod) (wfv1.NodePhase, string) {
	var newPhase wfv1.NodePhase
	var message string
	logger := logging.RequireLoggerFromContext(ctx)
	logger.WithField("namespace", pod.Namespace).
		WithField("podName", pod.Name).
		Info(ctx, "assessAgentPodStatus")
	switch pod.Status.Phase {
	case apiv1.PodSucceeded, apiv1.PodRunning, apiv1.PodPending:
		return "", ""
	case apiv1.PodFailed:
		newPhase = wfv1.NodeFailed
		message = pod.Status.Message
	default:
		newPhase = wfv1.NodeError

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the agent pod (`kubectl describe pod <agent-pod> -n <wf-namespace>`) to find the underlying reason (OOMKilled, Evicted, image error) and fix that cause
2. Retry the workflow — agent pod failures are usually transient infra issues, not workflow definition problems
3. If OOM, increase controller resource requests/limits for agent pods or reduce concurrency of HTTP/plugin tasks
4. If image pull issues, verify the argoexec image is available in the cluster/registry

// controller-side: nodes are already marked errored; on the client side handle with retry
try {
    await wf.wait()
} catch (e) {
    if (/agent pod failed/.test(e.message)) retryWorkflow(wfName)
}

When it happens

Trigger: The agent pod's container exits non-zero or is evicted/OOMKilled while HTTP template or executor-plugin tasks are in flight; reconcileAgentPod reads the updated pod and calls updateAgentPodStatus, which marks all WorkflowTaskSet nodes errored with the pod's reason message.

Common situations: Agent pod OOM-killed due to many concurrent HTTP actions; node pressure or eviction; the agent image failing to start (image pull error, RBAC missing for plugin connections); cluster autoscaler removing the agent node mid-run.

Related errors

• failed to read container args file %s: %w
• failed to unmarshal container args: %w
• failed to read template: %w
• failed to start command: %w
• failed to create emissary: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/f195d9d9d14558bb. Report an issue: GitHub.