What causes "failed to copy %s to %s: %w" in argoproj/argo-workflows?

Ensure the script finishes writing the parameter file before the main process exits (write atomically: write temp file then rename).

argoproj/argo-workflows · error

failed to copy to

Error message

failed to copy %s to %s: %w

What it means

saveParameter failed while copying the parameter source file's contents to the destination file with io.Copy. The parameter file was opened and created, but the transfer failed mid-way — typically a read error on src or a write/ENOSPC error on dst. The destination file may be partially written and the parameter will be missing or corrupt.

Solutions

Ensure the script finishes writing the parameter file before the main process exits (write atomically: write temp file then rename).
Keep parameter values small — use artifacts for large outputs; free disk space if ENOSPC.
Retry the step; persistent errors point to node storage problems — investigate/move workloads.
Verify no other process in the pod deletes or truncates the parameter path during execution.

Example fix

// before: in-place writes can be observed mid-write
printf 'partial'; printf ' value' >> /tmp/out
// after: atomic write so the reader never sees a partial file
printf value > /tmp/out.tmp && mv /tmp/out.tmp /tmp/out

Defensive patterns

Strategy: try-catch

Validate before calling

// in the step script: write parameters atomically and keep them small
// size check before declaring a parameter from a file
if fi, err := os.Stat(paramPath); err == nil && fi.Size() > 256*1024 {
    return errors.New("parameter too large; use an artifact instead")
}

Try / catch

err := runStep(ctx)
if err != nil && strings.Contains(err.Error(), "failed to copy") {
    // src mutated or ENOSPC mid-copy: retry the step; make scripts write atomically
    return retryWithBackoff(ctx, err)
}

Prevention

Write parameter files atomically (temp file + rename) in step scripts
Don't truncate or rotate the parameter file while the step runs
Use artifacts, not parameters, for large outputs
Add retryStrategy on steps so transient I/O failures self-heal

When it happens

Trigger: io.Copy(dst, src) returns error: the source file shrank/was deleted mid-read, src is a special file that errors on read, dst hits ENOSPC during write, or underlying storage I/O error.

Common situations: Script truncates/rotates the output file while the executor reads it; disk fills while writing large parameter values (parameters should be small); failing node storage.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/6f5e96c6fcd75fb3. Report an issue: GitHub.

Appendix: source

Thrown at cmd/argoexec/commands/emissary.go:756

		return fmt.Errorf("failed to open %s: %w", srcPath, err)
	}
	defer func() { _ = src.Close() }()
	dstPath := varRunArgo + "/outputs/parameters/" + srcPath
	logger.WithFields(logging.Fields{
		"src": srcPath,
		"dst": dstPath,
	}).Info(ctx, "saving parameter")
	z := filepath.Dir(dstPath)
	if mkdirErr := os.MkdirAll(z, 0o755); mkdirErr != nil { // chmod rwxr-xr-x
		return fmt.Errorf("failed to create directory %s: %w", z, mkdirErr)
	}
	dst, err := os.Create(dstPath)
	if err != nil {
		return fmt.Errorf("failed to create %s: %w", srcPath, err)
	}
	defer func() { _ = dst.Close() }()
	if _, err = io.Copy(dst, src); err != nil {
		return fmt.Errorf("failed to copy %s to %s: %w", srcPath, dstPath, err)
	}
	if err = dst.Close(); err != nil {
		return fmt.Errorf("failed to close %s: %w", dstPath, err)
	}
	return nil
}

View on GitHub (pinned to 35bff19146)