{"record":{"id":"51ee793451dd4775","repo":"argoproj/argo-workflows","slug":"writer-close-w","errorCode":null,"errorMessage":"writer close: %w","messagePattern":"writer close: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/artifacts/gcs/gcs.go","lineNumber":326,"sourceCode":"\n// upload an object to GCS\nfunc uploadObject(ctx context.Context, client *storage.Client, bucket, key, localPath string) error {\n\tf, err := os.Open(filepath.Clean(localPath))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"os open: %w\", err)\n\t}\n\tdefer func() {\n\t\tif closeErr := f.Close(); closeErr != nil {\n\t\t\tlogger := logging.RequireLoggerFromContext(ctx)\n\t\t\tlogger.WithField(\"path\", localPath).WithError(closeErr).Error(ctx, \"Error closing file\")\n\t\t}\n\t}()\n\twc := client.Bucket(bucket).Object(key).NewWriter(ctx)\n\tif _, err = io.Copy(wc, f); err != nil {\n\t\treturn fmt.Errorf(\"io copy: %w\", err)\n\t}\n\tif err := wc.Close(); err != nil {\n\t\treturn fmt.Errorf(\"writer close: %w\", err)\n\t}\n\treturn nil\n}\n\n// delete an object from GCS\nfunc deleteObject(ctx context.Context, client *storage.Client, bucket, key string) error {\n\terr := client.Bucket(bucket).Object(key).Delete(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"delete %s: %w\", key, err)\n\t}\n\treturn nil\n}\n\n// Delete deletes an artifact from GCS\nfunc (h *ArtifactDriver) Delete(ctx context.Context, s *wfv1.Artifact) error {\n\terr := waitutil.Backoff(defaultRetry,\n\t\tfunc() (bool, error) {\n\t\t\tclient, err := h.newGCSClient(ctx)","sourceCodeStart":308,"sourceCodeEnd":344,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/artifacts/gcs/gcs.go#L308-L344","documentation":"This error is thrown when storage.Writer.Close fails, i.e. finalizing the GCS multipart/resumable upload after io.Copy completed. In cloud.storage, Close is what actually flushes the buffered data and commits the object — an error here means the object was NOT successfully created, even though all bytes were handed to the writer. The wrapped error is typically the HTTP response error from GCS (e.g. 403, 429, 5xx) or a context cancellation.","triggerScenarios":"wc.Close() returns a non-nil error: the GCS API rejected the finalize request — insufficient IAM permissions (storage.objects.create denied), bucket versioning/retention policy conflicts, quota (rate limit 429), invalid object metadata, or the context was canceled between copy completion and finalize.","commonSituations":"Service account missing storage.objects.create while having read access; hitting GCS default per-bucket write rate limits with many parallel artifact uploads; bucket CMEK key revoked; pod terminated right at the end of upload (ctx canceled); object name violating bucket naming/retention rules.","solutions":["Read the wrapped error: a googleapi.Error code tells you the exact cause (403=IAM, 429=quota, 503=retry later).","Grant the artifact repository service account roles/storage.objectCreator (or objectAdmin) on the bucket: `gsutil iam ch serviceAccount:<sa>:roles/storage.objectCreator gs://<bucket>`.","Retry — the driver's backoff (waitutil.Backoff + isTransientGCSErr) retries transient 5xx/429; persistent 403 requires the IAM fix.","Check the bucket's CMEK/retention configuration if uploads to other buckets succeed.","Avoid pod termination racing the finalize: raise activeDeadlineSeconds or graceful termination period for artifact-heavy workflows."],"exampleFix":"// before: writer finalized without handling context cancel nicely\nwc := client.Bucket(bucket).Object(key).NewWriter(ctx)\n_, err = io.Copy(wc, f)\nif err := wc.Close(); err != nil {\n    return fmt.Errorf(\"writer close: %w\", err)\n}\n// after: set retries on the writer so transient finalize errors are handled client-side\nwc := client.Bucket(bucket).Object(key).NewWriter(ctx)\nwc.Retry = storage.RetryErrorInfo{ShouldRetry: func(err error) bool {\n    var ge *googleapi.Error\n    return errors.As(err, &ge) && (ge.Code == 429 || ge.Code >= 500)\n}}\n_, err = io.Copy(wc, f)\nif err := wc.Close(); err != nil {\n    return fmt.Errorf(\"writer close: %w\", err)\n}","handlingStrategy":"retry","validationCode":"// verify write permission before uploading\nbucketHandle := client.Bucket(bucket)\nif _, err := bucketHandle.IAM().TestPermissions(ctx,\n    []string{\"storage.objects.create\"}); err != nil {\n    return fmt.Errorf(\"missing storage.objects.create on %s: %w\", bucket, err)\n}","typeGuard":"func isFinalizeErr(err error) (*googleapi.Error, bool) {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) && strings.Contains(err.Error(), \"writer close:\") {\n        return gerr, true\n    }\n    return nil, false\n}","tryCatchPattern":"err := driver.Save(ctx, path, artifact)\nif err != nil {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) {\n        switch gerr.Code {\n        case 403:\n            // fix IAM: grant storage.objectCreator\n        case 429, 503:\n            // rate limit / transient: retry with exponential backoff\n        default:\n            // inspect gerr.Message for bucket policy/CMEK issues\n        }\n    }\n}","preventionTips":["Grant the artifact service account storage.objectCreator (or objectAdmin) and verify with IAM TestPermissions.","Enable client-side retries on storage.Writer (wc.Retry) for 429/5xx during finalize.","Throttle parallel artifact uploads to avoid per-bucket write rate limits.","Verify CMEK keys and retention policies on the bucket before large artifact runs.","Give workflows enough termination grace period so Close isn't cut off by pod shutdown."],"tags":["gcs","upload","writer-close","permissions"],"backgroundTag":"gcs-upload-finalize-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}