argoproj/argo-workflows · error

io copy: %w

Error message

io copy: %w

What it means

downloadObject streams the GCS object to the local file with io.Copy(out, rc). This error wraps any mid-transfer failure: the connection to GCS dropped, the context was cancelled, or reading the object body failed after the reader was opened.

Source

Thrown at workflow/artifacts/gcs/gcs.go:186

		if errors.Is(err, storage.ErrObjectNotExist) {
			return argoerrors.New(argoerrors.CodeNotFound, err.Error())
		}
		return fmt.Errorf("new bucket reader: %w", err)
	}
	defer rc.Close()
	out, err := os.Create(localPath)
	if err != nil {
		return fmt.Errorf("os create %s: %w", localPath, err)
	}
	defer func() {
		if closeErr := out.Close(); closeErr != nil {
			logger := logging.RequireLoggerFromContext(ctx)
			logger.WithField("path", localPath).WithError(closeErr).Error(ctx, "Error closing file")
		}
	}()
	_, err = io.Copy(out, rc)
	if err != nil {
		return fmt.Errorf("io copy: %w", err)
	}
	return nil
}

// list all the object names of the prefix in the bucket
func listByPrefix(ctx context.Context, client *storage.Client, bucket, prefix, delim string) ([]string, error) {
	ctx, cancel := context.WithTimeout(ctx, time.Second*30)
	defer cancel()
	it := client.Bucket(bucket).Objects(ctx, &storage.Query{
		Prefix:    prefix,
		Delimiter: delim,
	})
	results := []string{}
	for {
		attrs, err := it.Next()
		if errors.Is(err, iterator.Done) {
			break
		}

View on GitHub (pinned to 35bff19146)

Solutions

1. Retry the workflow or add retryStrategy to the step for transient network failures
2. Check/increase ephemeral-storage requests/limits so the download isn't killed
3. Investigate node networking (NetworkPolicy, Private Google Access, NAT timeouts) if it recurs
4. Use archive tar chunks or smaller artifacts to reduce single-transfer size

Example fix

// before
steps:
- - name: download
    template: get-artifact   # flaky, no retry
// after
- - name: download
    template: get-artifact
    retryStrategy: {limit: 3, retryPolicy: "TransientError"}

// pre-flight connectivity check from the pod network:
// kubectl run net-test --image=curlimages/curl --rm -it -- \
//   curl -sS -o /dev/null -w '%{http_code}' https://storage.googleapis.com

backoff := wait.Backoff{Steps: 3, Duration: time.Second, Factor: 2}
err := wait.ExponentialBackoff(backoff, func() (bool, error) {
	err := download(ctx, art, path)
	if err != nil && utilerrors.IsTransientErr(ctx, err) {
		return false, nil // retry
	}
	return err == nil, err
})

When it happens

Trigger: io.Copy fails during artifact download: network interruption to storage.googleapis.com, context deadline exceeded on large objects, GCS returning a resumable-read error, or disk filling up during the copy (write error surfaced through the copy).

Common situations: Large multi-GB artifacts over unstable networks; executor pod killed/OOM during copy; NetworkPolicy or NAT idle timeouts killing long-lived connections; exceeding ephemeral-storage limits mid-write.

Related errors

• new bucket reader: %w
• failed to stream artifact: %v
• failed to get and store artifact data: %w
• failed to read webhook request body: %w
• failed to close temp file: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/3bfddc4234a918c3. Report an issue: GitHub.