argoproj/argo-workflows · error

failed to write large arg %d to file: %w

Error message

failed to write large arg %d to file: %w

What it means

After loading args, emissary checks each arg: any single argument longer than common.MaxEnvVarLen is written to /tmp/argo_arg_<i>.txt and replaced in argv with the literal `@<path>` (the downstream program must support @filename syntax). This error is thrown when os.WriteFile to that temp path fails — typically no space, a read-only /tmp, or a permission issue — so the oversized arg cannot be offloaded and exec would otherwise blow the argument-size limit.

Source

Thrown at cmd/argoexec/commands/emissary.go:122

		argsData, readErr := os.ReadFile(argsFile)
		if readErr != nil {
			return fmt.Errorf("failed to read container args file %s: %w", argsFile, readErr)
		}
		var fileArgs []string
		if err = json.Unmarshal(argsData, &fileArgs); err != nil {
			return fmt.Errorf("failed to unmarshal container args: %w", err)
		}
		args = append(args, fileArgs...)
		logger.WithField("count", len(fileArgs)).Info(ctx, "Loaded container args from file")

		// Check for a large args and offload to file if needed
		// This avoids the exec() "argument list too long" error
		// Downstream programs should support @filename for parsing large args
		for i := 0; i < len(args); i++ {
			if len(args[i]) > common.MaxEnvVarLen {
				filePath := fmt.Sprintf("/tmp/argo_arg_%d.txt", i)
				if err = os.WriteFile(filePath, []byte(args[i]), 0o644); err != nil {
					return fmt.Errorf("failed to write large arg %d to file: %w", i, err)
				}
				logger.WithFields(logging.Fields{
					"argIndex": i,
					"size":     len(args[i]),
					"filePath": filePath,
				}).Info(ctx, "Offloaded large argument to file. Downstream program must support @filename syntax")
				args[i] = "@" + filePath
			}
		}
	}

	// In init-less pod mode the supervisor, not an init container, writes
	// /var/run/argo/template. Supervisor and main start concurrently, so
	// block until supervisor signals readiness (or failure) before reading
	// the template. Gated on an env var so legacy pods are unaffected.
	waitForReady := os.Getenv(common.EnvVarWaitForReady) == "true"
	if waitForReady {
		if waitErr := waitForSupervisorReady(ctx); waitErr != nil {

View on GitHub (pinned to 35bff19146)

Solutions

1. Free space / raise the sizeLimit of the volume mounted at /tmp (emptyDir sizeLimit or the node's disk).
2. Ensure the container's securityContext allows writing /tmp (run as a user with write access; /tmp mode 1777).
3. Reduce the size of the offending argument — e.g. pass it via an artifact or mounted file instead of as a container arg.
4. Have the downstream program consume the @filename indirection (emissary replaces the arg with `@/tmp/argo_arg_<i>.txt`); if it does not, restructure the workflow.
5. Remove stale /tmp/argo_arg_*.txt files from the image or an init step if they could collide.

Example fix

// before: pod with read-only small /tmp
volumes:
  - name: tmp
    emptyDir: { sizeLimit: 1Mi }
containers:
  - securityContext: { readOnlyRootFilesystem: true }
// after: writable, adequately sized /tmp
volumes:
  - name: tmp
    emptyDir: { sizeLimit: 1Gi }
containers:
  - volumeMounts: [ { name: tmp, mountPath: /tmp } ]

// Pre-check writability and space of the offload dir before emissary runs:
func validateTmpWritable(dir string) error {
	probe := filepath.Join(dir, ".argo-write-probe")
	if err := os.WriteFile(probe, []byte("ok"), 0o644); err != nil {
		return fmt.Errorf("%s not writable: %w", dir, err)
	}
	return os.Remove(probe)
}

if err := os.WriteFile(filePath, []byte(args[i]), 0o644); err != nil {
	switch {
	case errors.Is(err, os.ErrPermission):
		log.Errorf("cannot write arg offload file (permissions/readonly fs): %v", err)
	case errors.Is(err, syscall.ENOSPC):
		log.Errorf("no space left for arg offload file: %v", err)
	}
	return err
}

When it happens

Trigger: /tmp inside the workflow container is full or a read-only mount; the container user lacks write permission on /tmp; an existing /tmp/argo_arg_<i>.txt is owned by another user and unwritable; diskPressure evictions/emptyDir quota exhausted.

Common situations: Workflows with huge script bodies or giant parameters passed as container args; containers running as a non-root user with a restrictive securityContext; pods mounting an emptyDir over /tmp with a small sizeLimit; nested tooling that also writes /tmp/argo_arg_*.txt and races the emissary.

Related errors

• failed to read container args file %s: %w
• failed to read template: %w
• failed to leave working directory before staging input artif
• failed to create parent directory for artifact %q at %s: %w
• failed to clear existing path for artifact %q at %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/92a11f504a4ce971. Report an issue: GitHub.