What causes "create agent pod failed with reason:"%w"" in argoproj/argo-workflows?

Check the wrapped inner error in the controller log ('error in agent pod reconciliation') for the root cause

create agent pod failed with reason:"%w"

Error message

create agent pod failed with reason:"%w"

What it means

Wraps any error from reconcileAgentPod() during taskSet reconciliation and records it on all TaskSet-backed nodes via markTaskSetNodesError. The controller could not create or update the agent pod (the pod that runs HTTP templates and plugin executors), so dependent nodes are marked errored and the reconcile round aborts.

Source

Thrown at workflow/controller/taskset.go:108

func (woc *wfOperationCtx) getWorkflowTaskSet() (*wfv1.WorkflowTaskSet, error) {
	taskSet, exists, err := woc.controller.wfTaskSetInformer.Informer().GetIndexer().GetByKey(woc.wf.Namespace + "/" + woc.wf.Name)
	if err != nil {
		return nil, err
	}
	if !exists {
		return nil, nil
	}
	return taskSet.(*wfv1.WorkflowTaskSet), nil
}

func (woc *wfOperationCtx) taskSetReconciliation(ctx context.Context) {
	if err := woc.reconcileTaskSet(ctx); err != nil {
		woc.log.WithError(err).Error(ctx, "error in workflowtaskset reconciliation")
		return
	}
	if err := woc.reconcileAgentPod(ctx); err != nil {
		woc.log.WithError(err).Error(ctx, "error in agent pod reconciliation")
		woc.markTaskSetNodesError(ctx, fmt.Errorf(`create agent pod failed with reason:"%w"`, err))
		return
	}
}

func (woc *wfOperationCtx) nodeRequiresTaskSetReconciliation(ctx context.Context, nodeName string) bool {
	node, err := woc.wf.GetNodeByName(nodeName)
	if err != nil {
		return false
	}
	// If this node is of type HTTP, it will need an HTTP reconciliation
	if node.IsTaskSetNode() {
		return true
	}
	for _, child := range node.Children {
		// If any of the node's children need an HTTP reconciliation, the parent node will also need one
		childNodeName, err := woc.wf.Status.Nodes.GetName(child)
		if err != nil {
			woc.log.WithField("nodeID", child).WithFatal().Error(ctx, "was unable to get child node name for nodeID")

View on GitHub (pinned to 35bff19146)

Solutions

Check the wrapped inner error in the controller log ('error in agent pod reconciliation') for the root cause
Verify the workflow's service account has create/patch on pods in the namespace
Inspect the agent pod events (kubectl describe pod) for scheduling/quota failures
Validate the HTTP/plugin template spec with argo lint

Defensive patterns

Strategy: try-catch

Validate before calling

// preflight: ensure SA can create pods
kubectl auth can-i create pods --as=system:serviceaccount:<ns>:<sa> -n <ns>

Try / catch

// controller-side (Go)
if err := woc.reconcileAgentPod(ctx); err != nil {
    woc.log.WithError(err).Error(ctx, "error in agent pod reconciliation")
    woc.markTaskSetNodesError(ctx, fmt.Errorf(`create agent pod failed with reason:"%w"`, err))
    return
}

Prevention

Run argo lint on templates using HTTP/plugin executors
Grant the workflow SA pod create/patch RBAC
Watch namespace quotas and LimitRanges
Keep the WorkflowTaskSet CRD in sync with the controller version

When it happens

Trigger: reconcileAgentPod returns an error while creating/patching the agent pod: invalid pod spec, RBAC denial on pod create, quota exceeded, or the underlying mergePatch/create call failing during taskSetReconciliation triggered from executeWfLifeCycleHook or operate.

Common situations: HTTP/plugin templates with bad executor configuration; service account lacking pod-create permissions in the namespace; resource quotas or LimitRanges rejecting the agent pod spec; invalid template fields that only surface at pod-build time.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/cb8552c1d7ffd99e. Report an issue: GitHub.