argoproj/argo-workflows · error

artifact gc pod %q missing annotation %q

Error message

artifact gc pod %q missing annotation %q

What it means

When a completed artifact GC pod is reconciled, the controller reads the strategy from the pod's AnnotationKeyArtifactGCStrategy annotation to record per-strategy GC results. A completed pod missing this annotation cannot be attributed to a strategy, so this error is returned at workflow/controller/artifact_gc.go:642.

Source

Thrown at workflow/controller/artifact_gc.go:642

		}
		for _, a := range n.GetOutputs().GetArtifacts() {
			// artifact strategy is either based on overall Workflow ArtifactGC Strategy, or
			// if it's specified on the individual artifact level that takes priority
			artifactStrategy := woc.execWf.GetArtifactGCStrategy(&a)
			if artifactStrategy == strategy && !a.Deleted {
				results = append(results, wfv1.ArtifactSearchResult{Artifact: a, NodeID: n.ID})
			}
		}
	}
	return results
}

func (woc *wfOperationCtx) processCompletedArtifactGCPod(ctx context.Context, pod *corev1.Pod) error {
	woc.log.WithField("podName", pod.Name).Info(ctx, "processing completed Artifact GC Pod")

	strategyStr, found := pod.Annotations[common.AnnotationKeyArtifactGCStrategy]
	if !found {
		return fmt.Errorf("artifact gc pod %q missing annotation %q", pod.Name, common.AnnotationKeyArtifactGCStrategy)
	}
	strategy := wfv1.ArtifactGCStrategy(strategyStr)

	if pod.Status.Phase == corev1.PodFailed {
		errMsg := fmt.Sprintf("Artifact Garbage Collection failed for strategy %s, pod %s exited with non-zero exit code: check pod logs for more information", strategy, pod.Name)
		woc.addArtGCCondition(errMsg)
		woc.addArtGCEvent(errMsg)
	}

	// get associated WorkflowArtifactGCTasks
	labelSelector := fmt.Sprintf("%s = %s", common.LabelKeyArtifactGCPodHash, woc.artifactGCPodLabel(pod.Name))
	taskList, err := woc.controller.wfclientset.ArgoprojV1alpha1().WorkflowArtifactGCTasks(woc.wf.Namespace).List(ctx, metav1.ListOptions{LabelSelector: labelSelector})
	if err != nil {
		return fmt.Errorf("failed to List WorkflowArtifactGCTasks: %w", err)
	}

	for _, task := range taskList.Items {
		allArtifactsSucceeded, err := woc.processCompletedWorkflowArtifactGCTask(ctx, &task, strategy)

View on GitHub (pinned to 35bff19146)

Solutions

1. Delete the offending pod if it was manually created or is stale: kubectl -n <ns> delete pod <pod-name>.
2. If the annotation was stripped by manual edits, restore it (key: workflows.argoproj.io/artifact-gc-strategy) or delete and let the workflow re-create the GC pod.
3. Check for version skew: pods left over from a previous Argo version may carry different annotations; clean up leftover artifact-gc pods after upgrades.
4. Ensure no external controllers/mutators remove pod annotations in this namespace.

Example fix

// before: pod has component label but annotation removed by manual edit
// after: restore the annotation
kubectl -n my-ns annotate pod my-wf-artgc-wfcomp-12345 \
  workflows.argoproj.io/artifact-gc-strategy=OnWorkflowCompletion
// or simply delete the stale pod
kubectl -n my-ns delete pod my-wf-artgc-wfcomp-12345

// Before letting a workflow delete, verify its artifact-gc pods are well-formed
pods, _ := k8s.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{
    LabelSelector: "workflows.argoproj.io/component=artifact-gc," + "workflows.argoproj.io/workflow=" + wfName})
for _, p := range pods.Items {
    if p.Annotations["workflows.argoproj.io/artifact-gc-strategy"] == "" &&
       (p.Status.Phase == "Succeeded" || p.Status.Phase == "Failed") {
        // malformed pod — delete it so the workflow can recreate a correct one
    }
}

func hasGCStrategyAnnotation(pod *corev1.Pod) bool {
    _, ok := pod.Annotations[common.AnnotationKeyArtifactGCStrategy]
    return ok
}

if strings.Contains(err.Error(), "missing annotation") { /* delete the malformed pod or re-add the annotation, then requeue */ }

When it happens

Trigger: An artifact-gc pod (label components=artifact-gc / LabelKeyComponent) that succeeded or failed but lacks the artifactGCStrategy annotation — e.g. a manually created/edited pod mislabeled as artifact-gc, a pod created by an older controller version with different annotation keys, or an external tool stripping annotations.

Common situations: Manual kubectl edits to GC pods; users creating test pods with the artifact-gc component label but no annotations; upgrade scenarios where old-version pods used a different annotation name and are reconciled by a new controller.

Related errors

• can't process Artifact GC Strategy %s: template name %q belo
• can't find podInfo for podName %q
• ArtifactGCStrategy %q not valid
• failed to get WorkflowArtifactGCTask by key %q: %w
• failed to Create WorkflowArtifactGCTask %q for Garbage Colle

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/7d483ceaf79ba70b. Report an issue: GitHub.