argoproj/argo-workflows · error

WorkflowSpec may not change during execution when the contro

Error message

WorkflowSpec may not change during execution when the controller is set `templateReferencing: Secure`

What it means

In Secure template-referencing mode, the controller re-joins the user spec with the WorkflowTemplate's spec on every operation and compares it to the spec stored at submission. Any divergence means the effective spec would change mid-run, so reconciliation fails the workflow with this error.

Source

Thrown at workflow/controller/operator.go:4641

			return err
		}
		woc.wf.Status.StoredWorkflowSpec = &mergedWf.Spec
		woc.updated = true
	} else if woc.controller.Config.WorkflowRestrictions.MustNotChangeSpec() {
		wftHolder, err := woc.fetchWorkflowSpec(ctx)
		if err != nil {
			return err
		}
		userSpec := &woc.wf.Spec //nolint:forbidigo // not-woc-misuse
		if woc.controller.Config.WorkflowRestrictions.MustUseReference() {
			userSpec = wfutil.SanitizeUserWorkflowSpec(&woc.wf.Spec) //nolint:forbidigo // not-woc-misuse
		}
		mergedWf, err := wfutil.JoinWorkflowSpec(userSpec, wftHolder.GetWorkflowSpec(), &wfDefault.Spec)
		if err != nil {
			return err
		}
		if mergedWf.Spec.String() != woc.wf.Status.StoredWorkflowSpec.String() {
			return fmt.Errorf("WorkflowSpec may not change during execution when the controller is set `templateReferencing: Secure`")
		}
	}
	return nil
}

// mergedTemplateDefaultsInto modifies originalTmpl, setting any applicable default values.
func (woc *wfOperationCtx) mergedTemplateDefaultsInto(originalTmpl *wfv1.Template) error {
	if woc.execWf.Spec.TemplateDefaults == nil {
		return nil
	}

	originalTmplType := originalTmpl.GetType()
	applicableDefaults := woc.execWf.Spec.TemplateDefaults.DeepCopy()

	v := reflect.ValueOf(applicableDefaults).Elem()
	for i := 0; i < v.NumField(); i++ {
		field := v.Type().Field(i)
		// Check if the field is a pointer to a struct.

View on GitHub (pinned to 35bff19146)

Solutions

1. Restore the WorkflowTemplate content to what it was when the workflow was submitted
2. Do not modify the Workflow's spec after submission; resubmit a new workflow instead
3. Check for controller version changes in defaulting; pin/upgrade consistently and retry with a new workflow

// before editing, confirm no running workflow uses the template
kubectl get wf -n ns -o json | jq '.items[] | select(.status.storedWorkflowSpec != null)'

When it happens

Trigger: During reconciliation in templateReferencing: Secure mode, JoinWorkflowSpec(userSpec, templateSpec, defaults) produces a spec whose serialized form differs from Status.StoredWorkflowSpec.

Common situations: Someone edits the WorkflowTemplate while a workflow is running; a controller upgrade changes defaulting behavior so the merged output differs; manual kubectl edits to the running Workflow's spec.

Related errors

• failed to read container args file %s: %w
• failed to unmarshal container args: %w
• failed to read template: %w
• failed to start command: %w
• failed to create emissary: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/6b462a42df19c333. Report an issue: GitHub.