argoproj/argo-workflows · critical

cannot initialize a cached node from a non-memoized template

Error message

cannot initialize a cached node from a non-memoized template

What it means

Argo Workflows' memoization feature caches node results for templates that declare a `memoize:` section. `initializeCacheNode` (workflow/controller/operator.go:3005) is only ever supposed to be called for templates that participate in memoization; if the resolved template has `Memoize == nil`, the controller treats it as an unrecoverable internal invariant violation and panics instead of returning an error. This means the workflow-controller binary crashed (and the pod restarted) rather than just failing the node.

Source

Thrown at workflow/controller/operator.go:3005

	return nodeCtx, node
}

// initializeNodeOrMarkError initializes an error node or mark a node if it already exists.
func (woc *wfOperationCtx) initializeNodeOrMarkError(ctx context.Context, node *wfv1.NodeStatus, nodeName string, templateScope string, orgTmpl wfv1.TemplateReferenceHolder, boundaryID string, nodeFlag *wfv1.NodeFlag, err error) *wfv1.NodeStatus {
	if node != nil {
		return woc.markNodeError(ctx, nodeName, err)
	}

	_, n := woc.initializeNode(ctx, nodeName, wfv1.NodeTypeSkipped, templateScope, orgTmpl, boundaryID, wfv1.NodeError, nodeFlag, true, err.Error())
	return n
}

// Creates a node status that is or will be cached
// Returns the context with the node span and the node status.
func (woc *wfOperationCtx) initializeCacheNode(ctx context.Context, nodeName string, resolvedTmpl *wfv1.Template, templateScope string, orgTmpl wfv1.TemplateReferenceHolder, boundaryID string, memStat *wfv1.MemoizationStatus, nodeFlag *wfv1.NodeFlag, messages ...string) (context.Context, *wfv1.NodeStatus) {
	if resolvedTmpl.Memoize == nil {
		err := fmt.Errorf("cannot initialize a cached node from a non-memoized template")
		woc.log.WithFields(logging.Fields{"namespace": woc.wf.Namespace, "wfName": woc.wf.Name}).WithError(err)
		panic(err)
	}
	woc.log.WithFields(logging.Fields{
		"nodeName":       nodeName,
		"templateHolder": common.GetTemplateHolderString(orgTmpl),
		"boundaryID":     boundaryID,
	},
	).Debug(ctx, "Initializing cached node")

	nodeCtx, node := woc.initializeExecutableNode(ctx, nodeName, wfutil.GetNodeType(resolvedTmpl), templateScope, resolvedTmpl, orgTmpl, boundaryID, wfv1.NodePending, nodeFlag, false, messages...)
	node.MemoizationStatus = memStat
	return nodeCtx, node
}

// Creates a node status that has been cached, completely initialized, and marked as finished
// Returns the context with the node span and the node status.
func (woc *wfOperationCtx) initializeCacheHitNode(ctx context.Context, nodeName string, resolvedTmpl *wfv1.Template, templateScope string, orgTmpl wfv1.TemplateReferenceHolder, boundaryID string, outputs *wfv1.Outputs, memStat *wfv1.MemoizationStatus, nodeFlag *wfv1.NodeFlag, messages ...string) (context.Context, *wfv1.NodeStatus) {

View on GitHub (pinned to 35bff19146)

Solutions

1. Restore the `memoize:` block (with `key:` and `maxAge:`) on the template being referenced — the resolved template must be memoized for cached-node initialization
2. Do not edit/delete the ClusterWorkflowTemplate or WorkflowTemplate that running memoized workflows resolve against; delete and resubmit the affected workflow instead
3. If you are a developer: guard the caller with `if resolvedTmpl.Memoize == nil { return nil, errors.New(...) }` instead of relying on the panic, and ensure executeTemplate only computes memoization status when Memoize is set
4. Restart/recover the workflow-controller pod (kubectl rollout restart) if it crashed from this panic, then fix the spec before retrying

Example fix

// before (workflow spec)
templates:
- name: build          # memoize removed in an edit
  container:
    image: golang
// after
templates:
- name: build
  memoize:
    key: build-cache
    maxAge: "24h"
  container:
    image: golang

// validate the referenced template before submitting/updating
tmpl := resolvedTemplate
if tmpl.Memoize == nil && expectsMemoization {
    return fmt.Errorf("template %q must define memoize.key and memoize.maxAge", tmpl.Name)
}
// also: never mutate a ClusterWorkflowTemplate that running workflows memoize against

func isMemoizable(tmpl *wfv1.Template) bool {
    return tmpl != nil && tmpl.Memoize != nil && tmpl.Memoize.Key != ""
}

null

When it happens

Trigger: Reached only when the controller resolves a template reference (templateRef, template template, or steps/DAG entry) whose cached/memoization status is being initialized — e.g. a memoization key lookup path — but the resolved template ends up without `memoize:` set. Concrete causes: (1) a template ref points to a template that lost its `memoize:` block after the node ID was computed, so the stored node expects memoization but the new template doesn't have it; (2) a controller bug where `executeTemplate` computes `memStat` for a template it did not verify has `Memoize != nil`; (3) editing a workflow spec between retries/resubmits so a previously memoized node now resolves against a non-memoized template.

Common situations: Users upgrade Argo or edit a cluster workflow template and remove/rename the `memoize:` key while workflows referencing it are still running; users hand-edit submitted workflow specs; developers extending memoization to new template types call initializeCacheNode/initializeCacheHitNode with a template whose Memoize field was never validated.

Related errors

• invalid maxAge: %w
• single branch mode without a branch specified
• memoization configmap doesn't have %s label, refusing to use
• invalid cache key: %s
• duplicate synchronization item found

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/9013cf89544ec85e. Report an issue: GitHub.