argoproj/argo-workflows · warning

Not Found

Error message

Not Found

What it means

After a successful GET, the server validates the template against the configured instanceID via instanceIDService.Validate. On mismatch it intentionally returns a generic 404 'Not Found' without details, so outside callers cannot probe which templates exist in an instance. It looks identical to a missing template but the template actually exists.

Source

Thrown at server/artifacts/artifact_server.go:140

	allowed, err = auth.CanI(ctx, "create", "workflows", namespace, "")
	if err != nil {
		a.serverInternalError(ctx, err, w)
		return
	}
	if !allowed {
		http.Error(w, http.StatusText(http.StatusForbidden), http.StatusForbidden)
		return
	}

	// Get WorkflowTemplate to find artifact configuration
	wfClient := auth.GetWfClient(ctx)
	wfTemplate, err := wfClient.ArgoprojV1alpha1().WorkflowTemplates(namespace).Get(ctx, workflowTemplateName, metav1.GetOptions{})
	if err != nil {
		http.Error(w, fmt.Sprintf("Failed to get WorkflowTemplate %s/%s: %v", namespace, workflowTemplateName, err), http.StatusNotFound)
		return
	}
	if validateErr := a.instanceIDService.Validate(wfTemplate); validateErr != nil {
		http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
		return
	}

	// Enforce a hard cap on the total request body size before buffering any
	// of it, so a caller cannot exhaust server disk/memory with an oversized upload.
	maxUploadBytes, err := env.GetInt("ARGO_SERVER_MAX_ARTIFACT_UPLOAD_BYTES", 1<<30)
	if err != nil {
		a.serverInternalError(ctx, err, w)
		return
	}
	r.Body = http.MaxBytesReader(w, r.Body, int64(maxUploadBytes))

	// mime/multipart.ReadForm already removes temp files on parse error, but
	// registering cleanup here makes the handler's correctness independent of
	// that stdlib internal — any future error return still frees temp files.
	defer func() {
		if r.MultipartForm != nil {
			_ = r.MultipartForm.RemoveAll()

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the controller's instanceID setting in the workflow-controller configmap
2. Add the matching instanceID to the WorkflowTemplate metadata (or re-apply templates via the same instance's tooling)
3. Temporarily check argo-server logs to confirm the 404 is from instanceID validation rather than a missing object
4. If no multi-instance setup is intended, unset instanceID so validation passes for unlabeled templates

Example fix

# before
curl .../upload-artifacts/default/my-tmpl/art   # template exists, no instanceID label
# after
metadata:
  labels:
    workflows.argoproj.io/instance-id: my-instance  # match controller instanceID

tpl=$(kubectl get workflowtemplate <name> -n <ns> -o jsonpath='{.metadata.labels.workflows\.argoproj\.io/instance-id}')
ctrl=$(kubectl -n argo get cm workflow-controller-configmap -o yaml | grep instanceID)
[ "$tpl" = "${ctrl##*: }" ] || echo "instanceID mismatch"

if resp.StatusCode == 404 && templateExistsInCluster {
    // likely instanceID validation masking — compare labels vs controller config
}

When it happens

Trigger: ARGO_INSTANCEID (controller instanceID) is set but the WorkflowTemplate lacks the matching instanceID label/annotation (or vice versa) — validation fails and the handler masks it as 404.

Common situations: Multi-instance clusters (staging/prod controllers sharing a cluster) where templates were created before instanceID was configured or copied between namespaces without the label.

Related errors

• failed to validate workflow template instanceid: %w
• Failed to get WorkflowTemplate %s/%s: %v
• Artifact driver validation failed: The following artifact dr
• memoization configmap doesn't have %s label, refusing to use
• failed to get existing cluster workflow template %q to updat

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/e9d6e3b9cf55ee5d. Report an issue: GitHub.