argoproj/argo-workflows · error

failed to create pod: %w

Error message

failed to create pod: %w

What it means

The final step of createArtifactGCPod POSTs the constructed GC pod to the Kubernetes API. AlreadyExists is tolerated (idempotent reconcile), but any other error is wrapped at workflow/controller/artifact_gc.go:541 — commonly RBAC, quota, scheduling/admission, or invalid spec (e.g. a bad podSpecPatch).

Source

Thrown at workflow/controller/artifact_gc.go:541

		pod.Spec = *patchedPodSpec
	}

	// Use the Service Account and/or Labels and Annotations specified for our Pod, if they exist
	if info.serviceAccount != "" {
		pod.Spec.ServiceAccountName = info.serviceAccount
	}
	maps.Copy(pod.Labels, info.podMetadata.Labels)
	maps.Copy(pod.Annotations, info.podMetadata.Annotations)

	if v := woc.controller.Config.InstanceID; v != "" {
		pod.Labels[common.EnvVarInstanceID] = v
	}

	_, err = woc.controller.kubeclientset.CoreV1().Pods(woc.wf.Namespace).Create(ctx, pod, metav1.CreateOptions{})

	if err != nil {
		if !apierr.IsAlreadyExists(err) {
			return nil, fmt.Errorf("failed to create pod: %w", err)
		}
		woc.log.WithField("name", pod.Name).Warn(ctx, "Artifact GC Pod already exists")
	}
	return pod, nil
}

// go through any GC pods that are already running and may have completed
func (woc *wfOperationCtx) processArtifactGCCompletion(ctx context.Context) error {
	// check if any previous Artifact GC Pods completed
	pods, err := woc.controller.PodController.GetPodsByIndex(indexes.WorkflowIndex, woc.wf.GetNamespace()+"/"+woc.wf.GetName())
	if err != nil {
		return fmt.Errorf("failed to get pods from informer: %w", err)
	}

	for _, obj := range pods {
		pod := obj.(*corev1.Pod)
		if pod.Labels[common.LabelKeyComponent] != artifactGCComponent { // make sure it's an Artifact GC Pod
			continue

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped Kubernetes error: 403 -> fix RBAC for the controller; 422/Invalid -> check spec.artifactGC podSpecPatch syntax and PodSecurity admission labels on the namespace.
2. If quota exceeded (Cannot exceeded quota), raise ResourceQuota or free pods in the namespace.
3. Check any admission webhooks (Istio sidecar injection, Kyverno) failing/mutating the pod; exempt argo's artifact-gc pods if needed.
4. If the error stems from the podSpecPatch, run the patch through kubectl apply dry-run or fix ApplyPodSpecPatch input YAML.

Example fix

// before: podSpecPatch with wrong type
podSpecPatch: '{"containers":[{"name":"main","resources":{"limits":{"cpu": 100}}}']}'  // cpu must be string
// after
podSpecPatch: '{"containers":[{"name":"main","resources":{"limits":{"cpu": "100m"}}}]}'

// Validate podSpecPatch YAML/JSON before setting it in spec
var ps corev1.PodSpec
if err := json.Unmarshal([]byte(patch), &ps); err != nil { /* reject bad patch before submit */ }
// Check quota: kubectl -n ns describe resourcequota

func isPodCreateRBAC(err error) bool { return apierr.IsForbidden(err) && strings.Contains(err.Error(), "pods\"") }

if createErr != nil && !apierr.IsAlreadyExists(createErr) {
    switch {
    case apierr.IsForbidden(createErr): // fix RBAC
    case apierr.IsInvalid(createErr): // fix podSpecPatch or PodSecurity labels
    default: // quota/webhook issue — inspect wrapped error
    }
}

When it happens

Trigger: Pod Create rejected: controller service account lacks pods/create RBAC; ResourceQuota exceeded; PodSecurity admission (restricted) rejects the pod; mutating webhook failure; invalid podSpecPatch applied from artifact gc podSpecPatch config; image pull policy/config errors surfacing as spec validation.

Common situations: Clusters with restricted PodSecurityStandard where the GC pod's security context is non-compliant; namespace ResourceQuota blocking new pods; custom PodSpecPatch (workflow-level artifact gc metadata) with YAML type errors; namespace terminating.

Related errors

• failed to create Agent pod: %w
• failed to Create WorkflowArtifactGCTask %q for Garbage Colle
• failed to List WorkflowArtifactGCTasks: %w
• failed to get existing cluster workflow template %q to updat
• failed to list SSO RBAC service accounts: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/c9833bd196545f21. Report an issue: GitHub.