argoproj/argo-workflows · error

failed to close %s: %w

Error message

failed to close %s: %w

What it means

saveArtifact failed on the explicit dst.Close() after tarballing succeeded. The tar.gz data may not be fully flushed to disk, so the artifact file can be truncated or corrupt even though compression itself succeeded. This is uncommon and almost always a disk-level problem.

Source

Thrown at cmd/argoexec/commands/emissary.go:720

	dstPath := filepath.Join(varRunArgo, "/outputs/artifacts/", strings.TrimSuffix(srcPath, "/")+".tgz")
	logger.WithFields(logging.Fields{
		"src": srcPath,
		"dst": dstPath,
	}).Info(ctx, "saving artifact")
	z := filepath.Dir(dstPath)
	if err := os.MkdirAll(z, 0o755); err != nil { // chmod rwxr-xr-x
		return fmt.Errorf("failed to create directory %s: %w", z, err)
	}
	dst, err := os.Create(dstPath)
	if err != nil {
		return fmt.Errorf("failed to create destination %s: %w", dstPath, err)
	}
	defer func() { _ = dst.Close() }()
	if err = archive.TarGzToWriter(ctx, srcPath, gzip.DefaultCompression, dst); err != nil {
		return fmt.Errorf("failed to tarball the output %s to %s: %w", srcPath, dstPath, err)
	}
	if err = dst.Close(); err != nil {
		return fmt.Errorf("failed to close %s: %w", dstPath, err)
	}
	return nil
}

func saveParameter(ctx context.Context, template *wfv1.Template, srcPath string) error {
	logger := logging.RequireLoggerFromContext(ctx)

	if common.FindOverlappingVolume(template, srcPath) != nil {
		logger.WithField("src", srcPath).Info(ctx, "no need to save parameter - on overlapping volume")
		return nil
	}
	src, err := os.Open(filepath.Clean(srcPath))
	if os.IsNotExist(err) { // might be optional, so we ignore
		logger.WithField("src", srcPath).WithError(err).Warn(ctx, "cannot save parameter, does not exist")
		return nil
	}
	if err != nil {
		return fmt.Errorf("failed to open %s: %w", srcPath, err)

View on GitHub (pinned to 35bff19146)

Solutions

1. Retry the workflow step; if it recurs, investigate node storage health (dmesg, kubelet logs).
2. Free disk space — Close flushes remaining buffered gzip data and can hit ENOSPC.
3. Keep large artifacts on reliable volumes; avoid overfull local disks.
4. Check for truncated/corrupt artifact uploads from that node and cordon it if the I/O errors persist.

// pre-check storage health headroom
stat, err := os.Statfs("/var/run/argo/outputs")
if err == nil && stat.Bavail*uint64(stat.Bsize) < 100<<20 {
    return errors.New("less than 100MB free for artifact export")
}

err := runStep(ctx)
if err != nil && strings.Contains(err.Error(), "failed to close") {
    // close() flush failures are usually transient storage issues: retry once
    return retryOnce(ctx, err)
}

When it happens

Trigger: dst.Close() returns an error after TarGzToWriter: buffered data flush fails due to ENOSPC, I/O error on the underlying storage, or NFS/volume detach mid-write.

Common situations: Disk fills exactly during final flush of a large artifact; flaky network-attached storage (EBS/Ceph) returning I/O errors; node-level storage faults.

Related errors

• failed to create directory %s: %w
• failed to create destination %s: %w
• failed to stat input artifact %q at %s: %w
• failed to create parent directory for artifact %q at %s: %w
• failed to stat artifact path %q at %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/ebe566a13f203eda. Report an issue: GitHub.