can't process Artifact GC Strategy %s: node ID %q not found

Error message

can't process Artifact GC Strategy %s: node ID %q not found in Status

What it means

During Artifact GC, processArtifactGCStrategy groups artifact search results by pod and needs each artifact's originating node to derive its template. If the node ID recorded in the artifact search result cannot be found in workflow.Status.Nodes, the strategy cannot proceed and this error is returned, failing the GC reconciliation.

Source

Thrown at workflow/controller/artifact_gc.go:178

	for _, artifactSearchResult := range artifactSearchResults {
		// get the permissions required for this artifact and create a unique Pod name from them
		info = woc.getArtifactGCPodInfo(&artifactSearchResult.Artifact)
		podName, err = woc.artGCPodName(strategy, info)
		if err != nil {
			return err
		}
		if _, found := podNames[podName]; !found {
			podNames[podName] = info
		}
		if _, found := groupedByPod[podName]; !found {
			groupedByPod[podName] = make(templatesToArtifacts)
		}
		// get the Template for the Artifact
		node, nodeErr := woc.wf.Status.Nodes.Get(artifactSearchResult.NodeID)
		if nodeErr != nil {
			woc.log.WithField("nodeID", artifactSearchResult.NodeID).Error(ctx, "Was unable to obtain node")
			return fmt.Errorf("can't process Artifact GC Strategy %s: node ID %q not found in Status", strategy, artifactSearchResult.NodeID)
		}
		templateName := util.GetTemplateFromNode(*node)
		if templateName == "" {
			return fmt.Errorf("can't process Artifact GC Strategy %s: node %+v has an unnamed template", strategy, node)
		}
		template, found := templatesByName[templateName]
		if !found {
			template = woc.wf.GetTemplateByName(templateName)
			if template == nil {
				return fmt.Errorf("can't process Artifact GC Strategy %s: template name %q belonging to node %+v not found", strategy, templateName, node)
			}
			templatesByName[templateName] = template
		}

		if _, found := groupedByPod[podName][template.Name]; !found {
			groupedByPod[podName][template.Name] = make(wfv1.ArtifactSearchResults, 0)
		}

View on GitHub (pinned to 35bff19146)

Solutions

Inspect workflow Status.Nodes (`argo get <wf> -o yaml`) and compare with the nodeID in the artifact GC status
Remove stale artifact-gc-status entries or resubmit so GC state is rebuilt from current nodes
Disable aggressive node deletion (retention/podMetadata strategies) for workflows using artifactGC
Upgrade to a patched Argo version if status offloading is dropping node entries

Defensive patterns

Strategy: try-catch

Validate before calling

// before GC processing, verify node presence
if _, ok := wf.Status.Nodes[nodeID]; !ok {
    log.Warn("node missing from status, skipping artifact GC", nodeID)
}

Try / catch

err := garbageCollectArtifacts(ctx)
if err != nil && strings.Contains(err.Error(), "not found in Status") {
    // skip/rebuild GC state instead of failing the workflow
}

Prevention

Avoid retention policies that delete nodes for workflows using artifactGC
Don't hand-edit workflow Status.Nodes
Keep Argo upgraded for artifactGC/hydrator fixes

When it happens

Trigger: Status.Nodes no longer contains the node ID referenced by an artifact search result — e.g. status truncated/offloaded incorrectly, nodes pruned by workflow retention/pod-metadata deletion features, or corrupted/stale artifact GC status after a controller upgrade.

Common situations: Running artifactGC after the workflow's node status was compacted (podMetadata/retentionPolicy removed nodes); workflows retried/resubmitted so old node IDs persist in GC state; hydrator offloading bugs causing missing nodes on read.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/9f6516192ec1367b. Report an issue: GitHub.