argoproj/argo-workflows · error

upload %s: %w

Error message

upload %s: %w

What it means

Once the source is known to be a directory, uploadObjects uploads each contained file with uploadObject, wrapping per-file failures as "upload <path>". This error means one specific file inside the artifact directory failed to upload to GCS; the wrapped cause distinguishes the GCS API error (permission, quota, network) from local read errors.

Source

Thrown at workflow/artifacts/gcs/gcs.go:296

// upload a local file or dir to GCS
func uploadObjects(ctx context.Context, client *storage.Client, bucket, key, path string) error {
	isDir, err := file.IsDirectory(path)
	if err != nil {
		return fmt.Errorf("test if %s is a dir: %w", path, err)
	}
	if isDir {
		dirName := filepath.Clean(path) + string(os.PathSeparator)
		keyPrefix := filepath.Clean(key) + "/"
		fileRelPaths, listErr := listFileRelPaths(dirName, "")
		if listErr != nil {
			return listErr
		}
		for _, relPath := range fileRelPaths {
			fullKey := normalizeGCSKey(keyPrefix + relPath)
			err = uploadObject(ctx, client, bucket, fullKey, dirName+relPath)
			if err != nil {
				return fmt.Errorf("upload %s: %w", dirName+relPath, err)
			}
		}
	} else {
		objectKey := normalizeGCSKey(filepath.Clean(key))
		err = uploadObject(ctx, client, bucket, objectKey, path)
		if err != nil {
			return fmt.Errorf("upload %s: %w", path, err)
		}
	}
	return nil
}

// upload an object to GCS
func uploadObject(ctx context.Context, client *storage.Client, bucket, key, localPath string) error {
	f, err := os.Open(filepath.Clean(localPath))
	if err != nil {
		return fmt.Errorf("os open: %w", err)
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Grant the credential roles/storage.objectCreator (or objectAdmin) on the bucket
2. Check bucket encryption (CMEK) key permissions for the upload identity
3. Retry the workflow if the wrapped error is transient (5xx/network)
4. Inspect the directory for unreadable/special files and exclude them from the artifact path

Example fix

// before
gcloud projects remove-iam-policy-binding proj --member=serviceAccount:wf-sa@proj.iam --role=roles/storage.objectCreator
// after
gcloud projects add-iam-policy-binding proj --member=serviceAccount:wf-sa@proj.iam --role=roles/storage.objectCreator

// pre-flight write permission with the same identity:
// gcloud storage cp - / gs://my-bucket/.perm-check <<< "ok"
// fails with 403 if the identity lacks storage.objects.create

err := uploadArtifacts(ctx, dir)
if err != nil {
	if strings.Contains(err.Error(), "upload ") && utilerrors.IsTransientErr(ctx, err) {
		return retryWithBackoff(err) // 5xx/network per-file failure
	}
	return err // 403/quota: fix IAM, not retry
}

When it happens

Trigger: uploadObject fails for one file of a directory artifact: writer creation rejected by GCS (no storage.objects.create permission, bucket policy/quota), local file unreadable mid-listing (deleted between list and upload), or network failure during the upload stream.

Common situations: GSA missing roles/storage.objectCreator on the bucket; bucket CMEK/key permissions missing; huge output directories where one unreadable file (e.g. dangling symlink content, /proc-like entry) aborts the whole artifact save; transient 5xx from GCS.

Related errors

• new bucket reader: %w
• os create %s: %w
• test if %s is a dir: %w
• writer close: %w
• failed to create parent directory for artifact %q at %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/b431b6f222fc53c7. Report an issue: GitHub.