argoproj/argo-workflows · error

failed to validate workflow template instanceid

Error message

failed to validate workflow template instanceid: %w

What it means

After successfully fetching the referenced WorkflowTemplate, the dispatcher validates the template's `metadata.labels[workflows.argoproj.io/instance-id]` against the server's configured instance ID (via instanceid.Service.Validate). If the label is missing or does not match the controller's --instanceid, the template is considered owned by another Argo instance and dispatch is refused. This prevents one Argo installation from submitting workflows from templates it does not manage.

Solutions

1. Label the template with the server's instance ID: `kubectl label workflowtemplate <name> -n <ns> workflows.argoproj.io/instance-id=<value>` matching the controller's --instanceid flag.
2. Or create/manage the template through the same Argo instance (e.g. `argo submit --from workflowtemplate/<name>` via that instance) so the label is applied automatically.
3. If this Argo instance should accept all templates, unset --instanceid (empty instance ID accepts templates without the label).
4. Verify with `kubectl get wftmpl <name> -n <ns> --show-labels | grep workflows.argoproj.io/instance-id` that the value matches `kubectl -n argo get deploy workflow-controller -o yaml | grep instanceid`.

Example fix

# before: template missing instance-id label while server runs with --instanceid=prod
# after
kubectl label workflowtemplate my-template -n my-ns workflows.argoproj.io/instance-id=prod

# verify instance-id match before dispatch
SERVER_ID=$(kubectl -n argo get deploy workflow-controller -o jsonpath='{.spec.template.spec.containers[0].args[*]}' | grep -oP '(?<=--instanceid=)\S+')
TMPL_ID=$(kubectl get wftmpl <name> -n <ns> -o jsonpath='{.metadata.labels.workflows\.argoproj\.io/instance-id}')
[ "$SERVER_ID" = "$TMPL_ID" ] || kubectl label wftmpl <name> -n <ns> workflows.argoproj.io/instance-id=$SERVER_ID --overwrite

When it happens

Trigger: An event matched the binding and the template was fetched, but the template lacks the workflows.argoproj.io/instance-id label while the server runs with --instanceid set, or the label value differs (e.g. template created under a different controller instance or by plain `argo submit` on a differently configured cluster).

Common situations: Running multiple Argo controllers/instances against one cluster; templates copied from another namespace/cluster without the instance-id label; templates created manually with kubectl while the server uses a non-empty --instanceid; upgrading or migrating and forgetting to relabel templates.

Related errors

• 400
• 400
• 404
• agent pod failed with reason
• ArtifactGCStrategy not valid

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/c2d6af3dddea28af. Report an issue: GitHub.

Appendix: source

Thrown at server/event/dispatch/operation.go:114

	logger.WithFields(logging.Fields{"namespace": wfeb.Namespace, "event": wfeb.Name, "selector": selector, "matched": matched}).Debug(ctx, "Selector evaluation")
	submit := wfeb.Spec.Submit
	if matched && submit != nil {
		//nolint: contextcheck
		client := auth.GetWfClient(o.ctx)
		ref := wfeb.Spec.Submit.WorkflowTemplateRef
		var tmpl wfv1.WorkflowSpecHolder
		var err error
		if ref.ClusterScope {
			tmpl, err = client.ArgoprojV1alpha1().ClusterWorkflowTemplates().Get(ctx, ref.Name, metav1.GetOptions{})
		} else {
			tmpl, err = client.ArgoprojV1alpha1().WorkflowTemplates(wfeb.Namespace).Get(ctx, ref.Name, metav1.GetOptions{})
		}
		if err != nil {
			return nil, fmt.Errorf("failed to get workflow template: %w", err)
		}
		err = o.instanceIDService.Validate(tmpl)
		if err != nil {
			return nil, fmt.Errorf("failed to validate workflow template instanceid: %w", err)
		}
		wf := common.NewWorkflowFromWorkflowTemplate(tmpl.GetName(), ref.ClusterScope)

		// Apply workflowMetadata labels and annotations from the template
		// at creation time, matching the CronWorkflow behavior.
		// labelsFrom is left to the controller since it
		// requires parameter evaluation at runtime.
		if wmd := tmpl.GetWorkflowSpec().WorkflowMetadata; wmd != nil {
			maps.Copy(wf.Labels, wmd.Labels)
			if len(wmd.Annotations) > 0 {
				maps.Copy(wf.Annotations, wmd.Annotations)
			}
		}

		o.instanceIDService.Label(wf)
		err = o.populateWorkflowMetadata(wf, &submit.ObjectMeta)
		if err != nil {
			return nil, err

View on GitHub (pinned to 35bff19146)