argoproj/argo-workflows · error

io copy

Error message

io copy: %w

What it means

downloadObject streams the GCS object to the local file with io.Copy(out, rc). This error wraps any mid-transfer failure: the connection to GCS dropped, the context was cancelled, or reading the object body failed after the reader was opened.

Solutions

1. Retry the workflow or add retryStrategy to the step for transient network failures
2. Check/increase ephemeral-storage requests/limits so the download isn't killed
3. Investigate node networking (NetworkPolicy, Private Google Access, NAT timeouts) if it recurs
4. Use archive tar chunks or smaller artifacts to reduce single-transfer size

Example fix

// before
steps:
- - name: download
    template: get-artifact   # flaky, no retry
// after
- - name: download
    template: get-artifact
    retryStrategy: {limit: 3, retryPolicy: "TransientError"}

// pre-flight connectivity check from the pod network:
// kubectl run net-test --image=curlimages/curl --rm -it -- \
//   curl -sS -o /dev/null -w '%{http_code}' https://storage.googleapis.com

backoff := wait.Backoff{Steps: 3, Duration: time.Second, Factor: 2}
err := wait.ExponentialBackoff(backoff, func() (bool, error) {
	err := download(ctx, art, path)
	if err != nil && utilerrors.IsTransientErr(ctx, err) {
		return false, nil // retry
	}
	return err == nil, err
})

When it happens

Trigger: io.Copy fails during artifact download: network interruption to storage.googleapis.com, context deadline exceeded on large objects, GCS returning a resumable-read error, or disk filling up during the copy (write error surfaced through the copy).

Common situations: Large multi-GB artifacts over unstable networks; executor pod killed/OOM during copy; NetworkPolicy or NAT idle timeouts killing long-lived connections; exceeding ephemeral-storage limits mid-write.

Related errors

• new bucket reader
• artifact failed to load
• failed to check if key
• failed to close temp file
• failed to get and store artifact data

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/3bfddc4234a918c3. Report an issue: GitHub.

Appendix: source

Thrown at workflow/artifacts/gcs/gcs.go:186

		if errors.Is(err, storage.ErrObjectNotExist) {
			return argoerrors.New(argoerrors.CodeNotFound, err.Error())
		}
		return fmt.Errorf("new bucket reader: %w", err)
	}
	defer rc.Close()
	out, err := os.Create(localPath)
	if err != nil {
		return fmt.Errorf("os create %s: %w", localPath, err)
	}
	defer func() {
		if closeErr := out.Close(); closeErr != nil {
			logger := logging.RequireLoggerFromContext(ctx)
			logger.WithField("path", localPath).WithError(closeErr).Error(ctx, "Error closing file")
		}
	}()
	_, err = io.Copy(out, rc)
	if err != nil {
		return fmt.Errorf("io copy: %w", err)
	}
	return nil
}

// list all the object names of the prefix in the bucket
func listByPrefix(ctx context.Context, client *storage.Client, bucket, prefix, delim string) ([]string, error) {
	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
	defer cancel()
	it := client.Bucket(bucket).Objects(ctx, &storage.Query{
		Prefix:    prefix,
		Delimiter: delim,
	})
	results := []string{}
	for {
		attrs, err := it.Next()
		if errors.Is(err, iterator.Done) {
			break
		}

View on GitHub (pinned to 35bff19146)