argoproj/argo-workflows · critical

failed to unmarshal template: %w

Error message

failed to unmarshal template: %w

What it means

After successfully reading the template source (file or ARGO_TEMPLATE env), runEmissary json.Unmarshal's it into a wfv1.Template. This error is thrown when the bytes are not valid JSON or are not structurally a valid Template (wrong types, e.g. a string where an object is expected). Emissary cannot determine ContainerSet dependencies or input artifacts, so the pod fails before the user command runs.

Source

Thrown at cmd/argoexec/commands/emissary.go:160

			// to supervisor pre-main setup rather than the user command.
			// The process exit code (not just the exitcode file) must carry
			// the sentinel, because inferFailedReason keys off the container's
			// terminated exit code; wrap so main propagates 65 while keeping
			// waitErr's message.
			exitCode = common.ExitCodeSupervisorPreMainFailure
			logger.WithError(waitErr).Error(ctx, "supervisor failed before main container started")
			return argoerrors.NewExitErrWithCause(exitCode, waitErr)
		}
	}

	data, err := readTemplate()
	if err != nil {
		return fmt.Errorf("failed to read template: %w", err)
	}

	template := &wfv1.Template{}
	if err = json.Unmarshal(data, template); err != nil {
		return fmt.Errorf("failed to unmarshal template: %w", err)
	}

	// In init-less pod mode, main can't use the legacy per-artifact
	// SubPath bind mount (kubelet races the supervisor's write). The
	// input-artifacts volume is mounted whole at /argo/inputs/artifacts
	// and the emissary symlinks each input artifact into its expected
	// path once supervisor has finished writing (guaranteed by the
	// ready-marker wait above). Only `main` runs this — ContainerSet
	// children and sidecars don't get artifact paths symlinked in.
	if waitForReady && containerName == common.MainContainerName {
		if stageErr := stageInputArtifacts(ctx, template); stageErr != nil {
			// As above: propagate the sentinel as the process exit code so
			// inferFailedReason attributes this to supervisor pre-main setup.
			exitCode = common.ExitCodeSupervisorPreMainFailure
			logger.WithError(stageErr).Error(ctx, "failed to stage input artifacts before main container started")
			return argoerrors.NewExitErrWithCause(exitCode, stageErr)
		}
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Dump the actual bytes (`kubectl exec <pod> -c main -- cat /var/run/argo/template` or `echo $ARGO_TEMPLATE | base64 -d` if applicable) and validate the JSON.
2. Re-run the workflow to get a freshly written template; do not hand-edit the ConfigMap/offload data.
3. Align controller and executor versions (schema/marshal mismatches across upgrades).
4. Check controller logs for template env-offload errors (common.ResolveTemplateEnvValue path) if ARGO_TEMPLATE uses the offload sentinel.
5. If the template is legitimately huge, split it (smaller templates, fewer inline fields) to stay within env-var size limits.

Example fix

// before: ARGO_TEMPLATE truncated mid-object
ARGO_TEMPLATE='{"name":"main","inputs":{"artifacts":[{"name":"a'
// after: controller offloads oversized template to ConfigMap and sets sentinel,
// readTemplate resolves it via common.ResolveTemplateEnvValue
ARGO_TEMPLATE='offload:/argo/config/ARGO_TEMPLATE'

// Validate template bytes decode into wfv1.Template before relying on them:
func validateTemplateJSON(data []byte) error {
	var t wfv1.Template
	if err := json.Unmarshal(data, &t); err != nil {
		return fmt.Errorf("template payload invalid: %w", err)
	}
	if t.Name == "" {
		return fmt.Errorf("template payload decoded but has no name — wrong schema?")
	}
	return nil
}

func isJSONOrArray(b []byte) bool {
	var v any
	if err := json.Unmarshal(b, &v); err != nil {
		return false
	}
	switch v.(type) {
	case map[string]any, []any:
		return true
	}
	return false
}

template := &wfv1.Template{}
if err := json.Unmarshal(data, template); err != nil {
	var ute *json.UnmarshalTypeError
	if errors.As(err, &ute) {
		logger.WithError(err).Errorf(ctx, "template schema mismatch at %s (offset %d) — controller/executor version skew?", ute.Field, ute.Offset)
	}
	return fmt.Errorf("failed to unmarshal template: %w", err)
}

When it happens

Trigger: ARGO_TEMPLATE env value corrupted/truncated (exceeding env var limits, bad offload sentinel resolution); /var/run/argo/template partially written and read concurrently by emissary; a controller/executor version mismatch where one side marshals a schema the other cannot decode; the offloaded template chunk in the ConfigMap was modified or assembled out of order.

Common situations: Oversized templates offloaded to a ConfigMap and reassembled incorrectly after manual ConfigMap edits; upgrading Argo between versions with schema changes; custom templates injected via podspec patches that emit YAML not JSON; a race in a patched init-less layout reading the file while the supervisor is still writing it.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• failed to unmarshal container args: %w
• failed to read container args file %s: %w
• failed to write large arg %d to file: %w
• failed to read template: %w
• failed to start command: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/f8c0e2aad53b506f. Report an issue: GitHub.