argoproj/argo-workflows · critical

failed to unmarshal container args: %w

Error message

failed to unmarshal container args: %w

What it means

When container args were offloaded to a file (ARGO_CONTAINER_ARGS_FILE set), runEmissary reads the file and json.Unmarshal's it into []string. This error is thrown when the file contents are not a valid JSON array of strings. The args file is written by the controller as json.Marshal(container.Args), so invalid content means the offload contract between controller and executor is broken or the file is corrupted/truncated.

Source

Thrown at cmd/argoexec/commands/emissary.go:110

	// Note it's important varRunArgo+"/ctr/" folder is writable by all, because multiple containers may want to
	// write to it with different users.
	// This also indicates we've started.
	if err = os.MkdirAll(varRunArgo+"/ctr/"+containerName, 0o777); err != nil {
		return fmt.Errorf("failed to create ctr directory: %w", err)
	}

	name, args := args[0], args[1:]

	// Check if args were offloaded to a file (for large args that exceed exec limit)
	if argsFile := os.Getenv(common.EnvVarContainerArgsFile); argsFile != "" {
		logger.WithField("argsFile", argsFile).Info(ctx, "Reading container args from file")
		argsData, readErr := os.ReadFile(argsFile)
		if readErr != nil {
			return fmt.Errorf("failed to read container args file %s: %w", argsFile, readErr)
		}
		var fileArgs []string
		if err = json.Unmarshal(argsData, &fileArgs); err != nil {
			return fmt.Errorf("failed to unmarshal container args: %w", err)
		}
		args = append(args, fileArgs...)
		logger.WithField("count", len(fileArgs)).Info(ctx, "Loaded container args from file")

		// Check for a large args and offload to file if needed
		// This avoids the exec() "argument list too long" error
		// Downstream programs should support @filename for parsing large args
		for i := 0; i < len(args); i++ {
			if len(args[i]) > common.MaxEnvVarLen {
				filePath := fmt.Sprintf("/tmp/argo_arg_%d.txt", i)
				if err = os.WriteFile(filePath, []byte(args[i]), 0o644); err != nil {
					return fmt.Errorf("failed to write large arg %d to file: %w", i, err)
				}
				logger.WithFields(logging.Fields{
					"argIndex": i,
					"size":     len(args[i]),
					"filePath": filePath,
				}).Info(ctx, "Offloaded large argument to file. Downstream program must support @filename syntax")

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the file contents inside the pod (`kubectl exec <pod> -c <main> -- cat $ARGO_CONTAINER_ARGS_FILE`) and validate it is a JSON array of strings.
2. Re-run the workflow on a fresh pod: corrupted ConfigMaps are not repaired in place, a new pod gets a new one.
3. Align controller and executor versions — mismatched offload formats across upgrades are the usual cause.
4. Check the controller that wrote the ConfigMap (workflowpod.go) for a failed/partial write in its logs.
5. As a workaround, shrink the args below MaxEnvVarLen so args are passed via the pod spec directly, not the file.

Example fix

// before: file contains garbage
ARGO_CONTAINER_ARGS_FILE=/argo/staging/args.json -> "main" (not JSON)
// after: expected format written by the controller
["--flag","value"]

// Validate the offloaded args file parses as a JSON string array before use:
func validateArgsFile(path string) error {
	data, err := os.ReadFile(path)
	if err != nil {
		return err
	}
	var args []string
	if err := json.Unmarshal(data, &args); err != nil {
		return fmt.Errorf("args file %s is not a JSON string array: %w", path, err)
	}
	return nil
}

func isStringArray(v any) bool {
	arr, ok := v.([]any)
	if !ok {
		return false
	}
	for _, e := range arr {
		if _, ok := e.(string); !ok {
			return false
		}
	}
	return true
}

var fileArgs []string
if err := json.Unmarshal(argsData, &fileArgs); err != nil {
	var syntaxErr *json.SyntaxError
	switch {
	case errors.As(err, &syntaxErr):
		log.Errorf("malformed args JSON at offset %d: %v", syntaxErr.Offset, err)
	default:
		log.Errorf("args JSON type mismatch: %v", err)
	}
	return err
}

When it happens

Trigger: The file at ARGO_CONTAINER_ARGS_FILE contains non-JSON data (empty file, YAML, a placeholder, an error page), contains a JSON object/array of non-strings, or is truncated so the JSON does not parse.

Common situations: ConfigMap size limit (1MiB) truncation or failed write; someone manually edited or replaced the ConfigMap; a tool rewrote the ConfigMap data with different formatting; version skew where an older controller wrote a non-JSON format a newer executor cannot parse; the ConfigMap key holds the container name but the executor read the wrong key.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• failed to unmarshal template: %w
• failed to read container args file %s: %w
• failed to read template: %w
• failed to start command: %w
• neither %s nor %s is available

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/b5216cc9bbc215bd. Report an issue: GitHub.