{"record":{"id":"c9833bd196545f21","repo":"argoproj/argo-workflows","slug":"failed-to-create-pod-w","errorCode":null,"errorMessage":"failed to create pod: %w","messagePattern":"failed to create pod: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/controller/artifact_gc.go","lineNumber":541,"sourceCode":"\t\tpod.Spec = *patchedPodSpec\n\t}\n\n\t// Use the Service Account and/or Labels and Annotations specified for our Pod, if they exist\n\tif info.serviceAccount != \"\" {\n\t\tpod.Spec.ServiceAccountName = info.serviceAccount\n\t}\n\tmaps.Copy(pod.Labels, info.podMetadata.Labels)\n\tmaps.Copy(pod.Annotations, info.podMetadata.Annotations)\n\n\tif v := woc.controller.Config.InstanceID; v != \"\" {\n\t\tpod.Labels[common.EnvVarInstanceID] = v\n\t}\n\n\t_, err = woc.controller.kubeclientset.CoreV1().Pods(woc.wf.Namespace).Create(ctx, pod, metav1.CreateOptions{})\n\n\tif err != nil {\n\t\tif !apierr.IsAlreadyExists(err) {\n\t\t\treturn nil, fmt.Errorf(\"failed to create pod: %w\", err)\n\t\t}\n\t\twoc.log.WithField(\"name\", pod.Name).Warn(ctx, \"Artifact GC Pod already exists\")\n\t}\n\treturn pod, nil\n}\n\n// go through any GC pods that are already running and may have completed\nfunc (woc *wfOperationCtx) processArtifactGCCompletion(ctx context.Context) error {\n\t// check if any previous Artifact GC Pods completed\n\tpods, err := woc.controller.PodController.GetPodsByIndex(indexes.WorkflowIndex, woc.wf.GetNamespace()+\"/\"+woc.wf.GetName())\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get pods from informer: %w\", err)\n\t}\n\n\tfor _, obj := range pods {\n\t\tpod := obj.(*corev1.Pod)\n\t\tif pod.Labels[common.LabelKeyComponent] != artifactGCComponent { // make sure it's an Artifact GC Pod\n\t\t\tcontinue","sourceCodeStart":523,"sourceCodeEnd":559,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/artifact_gc.go#L523-L559","documentation":"The final step of createArtifactGCPod POSTs the constructed GC pod to the Kubernetes API. AlreadyExists is tolerated (idempotent reconcile), but any other error is wrapped at workflow/controller/artifact_gc.go:541 — commonly RBAC, quota, scheduling/admission, or invalid spec (e.g. a bad podSpecPatch).","triggerScenarios":"Pod Create rejected: controller service account lacks pods/create RBAC; ResourceQuota exceeded; PodSecurity admission (restricted) rejects the pod; mutating webhook failure; invalid podSpecPatch applied from artifact gc podSpecPatch config; image pull policy/config errors surfacing as spec validation.","commonSituations":"Clusters with restricted PodSecurityStandard where the GC pod's security context is non-compliant; namespace ResourceQuota blocking new pods; custom PodSpecPatch (workflow-level artifact gc metadata) with YAML type errors; namespace terminating.","solutions":["Read the wrapped Kubernetes error: 403 -> fix RBAC for the controller; 422/Invalid -> check spec.artifactGC podSpecPatch syntax and PodSecurity admission labels on the namespace.","If quota exceeded (Cannot exceeded quota), raise ResourceQuota or free pods in the namespace.","Check any admission webhooks (Istio sidecar injection, Kyverno) failing/mutating the pod; exempt argo's artifact-gc pods if needed.","If the error stems from the podSpecPatch, run the patch through kubectl apply dry-run or fix ApplyPodSpecPatch input YAML."],"exampleFix":"// before: podSpecPatch with wrong type\npodSpecPatch: '{\"containers\":[{\"name\":\"main\",\"resources\":{\"limits\":{\"cpu\": 100}}}']}'  // cpu must be string\n// after\npodSpecPatch: '{\"containers\":[{\"name\":\"main\",\"resources\":{\"limits\":{\"cpu\": \"100m\"}}}]}'","handlingStrategy":"try-catch","validationCode":"// Validate podSpecPatch YAML/JSON before setting it in spec\nvar ps corev1.PodSpec\nif err := json.Unmarshal([]byte(patch), &ps); err != nil { /* reject bad patch before submit */ }\n// Check quota: kubectl -n ns describe resourcequota","typeGuard":"func isPodCreateRBAC(err error) bool { return apierr.IsForbidden(err) && strings.Contains(err.Error(), \"pods\\\"\") }","tryCatchPattern":"if createErr != nil && !apierr.IsAlreadyExists(createErr) {\n    switch {\n    case apierr.IsForbidden(createErr): // fix RBAC\n    case apierr.IsInvalid(createErr): // fix podSpecPatch or PodSecurity labels\n    default: // quota/webhook issue — inspect wrapped error\n    }\n}","preventionTips":["Enforce restricted PodSecurity on workflow namespaces so GC pods pass admission.","Keep podSpecPatch JSON valid: resource quantities must be strings (\"100m\", \"64Mi\").","Ensure namespace ResourceQuota has headroom for GC pods.","Exempt artifact-gc pods from failing admission/mutating webhooks."],"tags":["kubernetes","pod-creation","rbac","artifact-gc","admission"],"backgroundTag":"pod-create-rejected","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}