argoproj/argo-workflows · error

writer close: %w

Error message

writer close: %w

What it means

This error is thrown when storage.Writer.Close fails, i.e. finalizing the GCS multipart/resumable upload after io.Copy completed. In cloud.storage, Close is what actually flushes the buffered data and commits the object — an error here means the object was NOT successfully created, even though all bytes were handed to the writer. The wrapped error is typically the HTTP response error from GCS (e.g. 403, 429, 5xx) or a context cancellation.

Source

Thrown at workflow/artifacts/gcs/gcs.go:326

// upload an object to GCS
func uploadObject(ctx context.Context, client *storage.Client, bucket, key, localPath string) error {
	f, err := os.Open(filepath.Clean(localPath))
	if err != nil {
		return fmt.Errorf("os open: %w", err)
	}
	defer func() {
		if closeErr := f.Close(); closeErr != nil {
			logger := logging.RequireLoggerFromContext(ctx)
			logger.WithField("path", localPath).WithError(closeErr).Error(ctx, "Error closing file")
		}
	}()
	wc := client.Bucket(bucket).Object(key).NewWriter(ctx)
	if _, err = io.Copy(wc, f); err != nil {
		return fmt.Errorf("io copy: %w", err)
	}
	if err := wc.Close(); err != nil {
		return fmt.Errorf("writer close: %w", err)
	}
	return nil
}

// delete an object from GCS
func deleteObject(ctx context.Context, client *storage.Client, bucket, key string) error {
	err := client.Bucket(bucket).Object(key).Delete(ctx)
	if err != nil {
		return fmt.Errorf("delete %s: %w", key, err)
	}
	return nil
}

// Delete deletes an artifact from GCS
func (h *ArtifactDriver) Delete(ctx context.Context, s *wfv1.Artifact) error {
	err := waitutil.Backoff(defaultRetry,
		func() (bool, error) {
			client, err := h.newGCSClient(ctx)

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped error: a googleapi.Error code tells you the exact cause (403=IAM, 429=quota, 503=retry later).
2. Grant the artifact repository service account roles/storage.objectCreator (or objectAdmin) on the bucket: `gsutil iam ch serviceAccount:<sa>:roles/storage.objectCreator gs://<bucket>`.
3. Retry — the driver's backoff (waitutil.Backoff + isTransientGCSErr) retries transient 5xx/429; persistent 403 requires the IAM fix.
4. Check the bucket's CMEK/retention configuration if uploads to other buckets succeed.
5. Avoid pod termination racing the finalize: raise activeDeadlineSeconds or graceful termination period for artifact-heavy workflows.

Example fix

// before: writer finalized without handling context cancel nicely
wc := client.Bucket(bucket).Object(key).NewWriter(ctx)
_, err = io.Copy(wc, f)
if err := wc.Close(); err != nil {
    return fmt.Errorf("writer close: %w", err)
}
// after: set retries on the writer so transient finalize errors are handled client-side
wc := client.Bucket(bucket).Object(key).NewWriter(ctx)
wc.Retry = storage.RetryErrorInfo{ShouldRetry: func(err error) bool {
    var ge *googleapi.Error
    return errors.As(err, &ge) && (ge.Code == 429 || ge.Code >= 500)
}}
_, err = io.Copy(wc, f)
if err := wc.Close(); err != nil {
    return fmt.Errorf("writer close: %w", err)
}

// verify write permission before uploading
bucketHandle := client.Bucket(bucket)
if _, err := bucketHandle.IAM().TestPermissions(ctx,
    []string{"storage.objects.create"}); err != nil {
    return fmt.Errorf("missing storage.objects.create on %s: %w", bucket, err)
}

func isFinalizeErr(err error) (*googleapi.Error, bool) {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) && strings.Contains(err.Error(), "writer close:") {
        return gerr, true
    }
    return nil, false
}

err := driver.Save(ctx, path, artifact)
if err != nil {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) {
        switch gerr.Code {
        case 403:
            // fix IAM: grant storage.objectCreator
        case 429, 503:
            // rate limit / transient: retry with exponential backoff
        default:
            // inspect gerr.Message for bucket policy/CMEK issues
        }
    }
}

When it happens

Trigger: wc.Close() returns a non-nil error: the GCS API rejected the finalize request — insufficient IAM permissions (storage.objects.create denied), bucket versioning/retention policy conflicts, quota (rate limit 429), invalid object metadata, or the context was canceled between copy completion and finalize.

Common situations: Service account missing storage.objects.create while having read access; hitting GCS default per-bucket write rate limits with many parallel artifact uploads; bucket CMEK key revoked; pod terminated right at the end of upload (ctx canceled); object name violating bucket naming/retention rules.

Related errors

• upload %s: %w
• unable to open file %s: %w
• new bucket reader: %w
• os create %s: %w
• test if %s is a dir: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/51ee793451dd4775. Report an issue: GitHub.