argoproj/argo-workflows · error

new bucket reader: %w

Error message

new bucket reader: %w

What it means

downloadObject opens a reader for the GCS object via client.Bucket(bucket).Object(objName).NewReader(ctx). Non-not-found failures are wrapped as "new bucket reader". A not-found object is converted to a coded CodeNotFound error instead. This error means the GCS API rejected the read request itself — auth, permission, bucket, or network issues.

Source

Thrown at workflow/artifacts/gcs/gcs.go:171

}

// download an object from the bucket
func downloadObject(ctx context.Context, client *storage.Client, bucket, key, objName, path string) error {
	objPrefix := normalizeGCSKey(filepath.Clean(key))
	relObjPath := strings.TrimPrefix(objName, objPrefix)
	localPath := filepath.Join(path, relObjPath)
	objectDir, _ := filepath.Split(localPath)
	if objectDir != "" {
		if err := os.MkdirAll(objectDir, 0o700); err != nil {
			return fmt.Errorf("mkdir %s: %w", objectDir, err)
		}
	}
	rc, err := client.Bucket(bucket).Object(objName).NewReader(ctx)
	if err != nil {
		if errors.Is(err, storage.ErrObjectNotExist) {
			return argoerrors.New(argoerrors.CodeNotFound, err.Error())
		}
		return fmt.Errorf("new bucket reader: %w", err)
	}
	defer rc.Close()
	out, err := os.Create(localPath)
	if err != nil {
		return fmt.Errorf("os create %s: %w", localPath, err)
	}
	defer func() {
		if closeErr := out.Close(); closeErr != nil {
			logger := logging.RequireLoggerFromContext(ctx)
			logger.WithField("path", localPath).WithError(closeErr).Error(ctx, "Error closing file")
		}
	}()
	_, err = io.Copy(out, rc)
	if err != nil {
		return fmt.Errorf("io copy: %w", err)
	}
	return nil
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Verify the bucket name in the artifact spec and that the credential has storage.objects.get (roles/storage.objectViewer or objectAdmin)
2. Check workload identity / secret key wiring so requests are authenticated
3. If transient, retry the workflow; add retryStrategy for flaky network conditions
4. Confirm egress/Private Google Access connectivity to storage.googleapis.com

Example fix

// before
kubectl annotate ksa default iam.gke.io/gcp-service-account-  # binding removed
// after
kubectl annotate ksa default iam.gke.io/gcp-service-account=artifact-gsa@project.iam.gserviceaccount.com
gcloud iam service-accounts add-iam-policy-binding artifact-gsa@project.iam.gserviceaccount.com --role=roles/iam.workloadIdentityUser --member=principal://...

// pre-flight permission check with the same credential:
// kubectl run test-read --serviceaccount=<ksa> --image=gcr.io/google.com/cloudsdktool/cloud-sdk -- \
//   gcloud storage objects describe gs://my-bucket/path/to/artifact

import "cloud.google.com/go/storage"
func isNotFound(err error) bool {
	return errors.Is(err, storage.ErrObjectNotExist)
}

err := download(ctx, art, path)
if err != nil {
	var coded argoerrors.CodedError
	if errors.As(err, &coded) && coded.Code() == argoerrors.CodeNotFound {
		return handleMissingArtifact() // not this error
	}
	if isTransient(ctx, err) { // util/errors.IsTransientErr
		return retry(err)
	}
	return err
}

When it happens

Trigger: NewReader fails with an error that is not storage.ErrObjectNotExist: wrong bucket name, no storage.objects.get permission for the credential, expired/invalid credentials, request cancelled by context timeout, or transient network failure to storage.googleapis.com.

Common situations: Typo in bucket name; GSA lacks roles/storage.objectViewer; workload identity not bound so the request is anonymous; VPC without Private Google Access trying to reach GCS; per-request ctx timeout too short for large objects.

Related errors

• os create %s: %w
• io copy: %w
• upload %s: %w
• failed to check if key %s exists from bucket %s: %w
• failed to stream artifact: %v

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/ba29ca5aeb6cc2f7. Report an issue: GitHub.