argoproj/argo-workflows · error

failed to List WorkflowArtifactGCTasks: %w

Error message

failed to List WorkflowArtifactGCTasks: %w

What it means

After an artifact GC pod completes, the controller lists WorkflowArtifactGCTasks labeled with the pod's FNV hash (LabelKeyArtifactGCPodHash) to evaluate per-artifact deletion results. Any List API failure is wrapped at workflow/controller/artifact_gc.go:656 — most often RBAC denial or API-server unavailability.

Source

Thrown at workflow/controller/artifact_gc.go:656

	woc.log.WithField("podName", pod.Name).Info(ctx, "processing completed Artifact GC Pod")

	strategyStr, found := pod.Annotations[common.AnnotationKeyArtifactGCStrategy]
	if !found {
		return fmt.Errorf("artifact gc pod %q missing annotation %q", pod.Name, common.AnnotationKeyArtifactGCStrategy)
	}
	strategy := wfv1.ArtifactGCStrategy(strategyStr)

	if pod.Status.Phase == corev1.PodFailed {
		errMsg := fmt.Sprintf("Artifact Garbage Collection failed for strategy %s, pod %s exited with non-zero exit code: check pod logs for more information", strategy, pod.Name)
		woc.addArtGCCondition(errMsg)
		woc.addArtGCEvent(errMsg)
	}

	// get associated WorkflowArtifactGCTasks
	labelSelector := fmt.Sprintf("%s = %s", common.LabelKeyArtifactGCPodHash, woc.artifactGCPodLabel(pod.Name))
	taskList, err := woc.controller.wfclientset.ArgoprojV1alpha1().WorkflowArtifactGCTasks(woc.wf.Namespace).List(ctx, metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return fmt.Errorf("failed to List WorkflowArtifactGCTasks: %w", err)
	}

	for _, task := range taskList.Items {
		allArtifactsSucceeded, err := woc.processCompletedWorkflowArtifactGCTask(ctx, &task, strategy)
		if err != nil {
			return err
		}
		if allArtifactsSucceeded && pod.Status.Phase == corev1.PodSucceeded {
			// now we can delete it, if it succeeded (otherwise we leave it up to be inspected)
			woc.log.WithField("name", task.Name).Debug(ctx, "deleting WorkflowArtifactGCTask")
			err := woc.controller.wfclientset.ArgoprojV1alpha1().WorkflowArtifactGCTasks(woc.wf.Namespace).Delete(ctx, task.Name, metav1.DeleteOptions{})
			if err != nil {
				woc.log.WithField("name", task.Name).WithError(err).Error(ctx, "error deleting WorkflowArtifactGCTask")
			}
		}
	}
	return nil
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the wrapped error: 403 -> grant list on workflowartifactgctasks to the workflow-controller service account (compare with manifests/install.yaml RBAC for your version).
2. For timeouts/connection errors, check API server health and controller-to-API-server network; the reconcile is requeued so transient failures self-heal.
3. Verify the CRD is installed and functioning: kubectl get workflowartifactgctasks -n <ns>.
4. If a proxy/webhook strips label selectors, exempt argoproj.io API traffic from it.

Example fix

// before: ClusterRole missing list on the GC task resource
// after
rules:
  - apiGroups: ["argoproj.io"]
    resources: ["workflowartifactgctasks"]
    verbs: ["list", "get", "watch", "delete"]

cmd := exec.Command("kubectl", "auth", "can-i", "list", "workflowartifactgctasks.argoproj.io",
    "-n", ns, "--as", "system:serviceaccount:argo:workflow-controller")

apierr.IsForbidden(err) // distinguishes RBAC from transient API failures

if listErr != nil {
    if apierr.IsForbidden(listErr) { /* fix RBAC permanently */ }
    // otherwise transient: controller requeue will retry automatically
}

When it happens

Trigger: GET /apis/argoproj.io/v1alpha1/.../workflowartifactgctasks?labelSelector=... fails: controller service account lacks list permission; API server timeout/network error; namespace terminating; aggregated admission or quota issues.

Common situations: RBAC tightened without workflowartifactgctasks/list; API server overload during large cluster events; label selector passing through a validating proxy that rejects it.

Related errors

• failed to Create WorkflowArtifactGCTask %q for Garbage Colle
• failed to create pod: %w
• failed to get existing cluster workflow template %q to updat
• failed to list SSO RBAC service accounts: %w
• failed to get service account secret: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/731a5843f202f777. Report an issue: GitHub.