{"record":{"id":"cfd6204b902f890d","repo":"argoproj/argo-workflows","slug":"delete-s-w","errorCode":null,"errorMessage":"delete %s: %w","messagePattern":"delete (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/artifacts/gcs/gcs.go","lineNumber":335,"sourceCode":"\t\t\tlogger := logging.RequireLoggerFromContext(ctx)\n\t\t\tlogger.WithField(\"path\", localPath).WithError(closeErr).Error(ctx, \"Error closing file\")\n\t\t}\n\t}()\n\twc := client.Bucket(bucket).Object(key).NewWriter(ctx)\n\tif _, err = io.Copy(wc, f); err != nil {\n\t\treturn fmt.Errorf(\"io copy: %w\", err)\n\t}\n\tif err := wc.Close(); err != nil {\n\t\treturn fmt.Errorf(\"writer close: %w\", err)\n\t}\n\treturn nil\n}\n\n// delete an object from GCS\nfunc deleteObject(ctx context.Context, client *storage.Client, bucket, key string) error {\n\terr := client.Bucket(bucket).Object(key).Delete(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"delete %s: %w\", key, err)\n\t}\n\treturn nil\n}\n\n// Delete deletes an artifact from GCS\nfunc (h *ArtifactDriver) Delete(ctx context.Context, s *wfv1.Artifact) error {\n\terr := waitutil.Backoff(defaultRetry,\n\t\tfunc() (bool, error) {\n\t\t\tclient, err := h.newGCSClient(ctx)\n\t\t\tif err != nil {\n\t\t\t\treturn !isTransientGCSErr(ctx, err), err\n\t\t\t}\n\t\t\tdefer client.Close()\n\t\t\terr = deleteObject(ctx, client, s.GCS.Bucket, s.GCS.Key)\n\t\t\tif err != nil {\n\t\t\t\treturn !isTransientGCSErr(ctx, err), err\n\t\t\t}\n\t\t\treturn true, nil","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/artifacts/gcs/gcs.go#L317-L353","documentation":"This error is thrown by deleteObject when the GCS API client's Object.Delete call fails while removing an artifact object from a bucket. It wraps the storage library error with the object key so the operator knows which object could not be deleted. ArtifactDriver.Delete retries it via waitutil.Backoff, so reaching the caller means the error persisted (or is non-transient, e.g. NotFound or PermissionDenied).","triggerScenarios":"client.Bucket(bucket).Object(key).Delete(ctx) returns an error: object does not exist (storage.ErrObjectNotExist / 404), service account lacks storage.objects.delete (403), bucket is versioned with retention lock, rate-limited (429), or the context was canceled (workflow deleted mid-cleanup).","commonSituations":"Deleting an artifact that was never uploaded (garbage-collection cleanup of failed workflows); IAM role downgraded to read-only on the artifact bucket; bucket-level retention policy or Object Lock preventing deletion; deleting already-pruned objects twice; transient GCS 5xx exhausting the retry backoff.","solutions":["Check the wrapped error code: 404 is usually harmless (object already gone) — consider treating storage.ErrObjectNotExist as success in cleanup paths.","Grant storage.objectAdmin (or objectDelete) on the bucket to the workflow's service account: `gsutil iam ch serviceAccount:<sa>:roles/storage.objectAdmin gs://<bucket>`.","Verify the key/bucket in the artifact record is correct — a stale or mis-encoded key can point at a nonexistent object.","Check bucket retention policy/lock: `gsutil retention ls gs://<bucket>`; wait out or lift the retention period if deletion is genuinely required.","Retry later if the wrapped error is 429/5xx; the driver's defaultRetry backoff may need more attempts for heavily throttled buckets."],"exampleFix":"// before: cleanup fails the workflow when the object is already gone\nerr = deleteObject(ctx, client, s.GCS.Bucket, s.GCS.Key)\nif err != nil {\n    return fmt.Errorf(\"delete %s: %w\", key, err)\n}\n// after: tolerate NotFound during cleanup\nerr = deleteObject(ctx, client, s.GCS.Bucket, s.GCS.Key)\nif err != nil {\n    if status.Code(err) == codes.NotFound {\n        return nil\n    }\n    return fmt.Errorf(\"delete %s: %w\", key, err)\n}","handlingStrategy":"try-catch","validationCode":"// check the object exists and is deletable before calling Delete\nhandle := client.Bucket(bucket).Object(key)\nif _, err := handle.Attrs(ctx); err != nil {\n    if status.Code(err) == codes.NotFound {\n        return nil // nothing to delete\n    }\n    return fmt.Errorf(\"cannot access gs://%s/%s: %w\", bucket, key, err)\n}","typeGuard":"func isDeleteWrapErr(err error) (*googleapi.Error, bool) {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) && strings.Contains(err.Error(), \"delete \") {\n        return gerr, true\n    }\n    return nil, false\n}","tryCatchPattern":"err := driver.Delete(ctx, artifact)\nif err != nil {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) {\n        switch gerr.Code {\n        case 404:\n            // object already gone: treat as success in cleanup paths\n        case 403:\n            // IAM: grant storage.objects.delete / objectAdmin\n        default:\n            // 429/5xx: backoff and retry\n        }\n    }\n}","preventionTips":["Grant the service account storage.objectAdmin on artifact buckets it must clean up.","Treat storage.ErrObjectNotExist (404) as success in idempotent cleanup/GC code paths.","Check bucket retention policies and Object Lock before relying on programmatic deletion.","Avoid deleting artifacts of running workflows; only clean up after terminal phases.","Log the bucket/key from the error message to detect stale artifact records pointing at wrong keys."],"tags":["gcs","delete","permissions","object-not-found"],"backgroundTag":"gcs-object-delete-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}