{"record":{"id":"88e9fd8c220b477e","repo":"kubernetes/kops","slug":"error-removing-file-s-w-88e9fd","errorCode":null,"errorMessage":"error removing file %s: %w","messagePattern":"error removing file (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/pkg/vfs/gsfs.go","lineNumber":155,"sourceCode":"\t\treturn err\n\t} else if done {\n\t\treturn nil\n\t} else {\n\t\t// Shouldn't happen - we always return a non-nil error with false\n\t\treturn wait.ErrWaitTimeout\n\t}\n}\n\nfunc (p *GSPath) RemoveAll(ctx context.Context) error {\n\ttree, err := p.ReadTree(ctx)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tfor _, objectPath := range tree {\n\t\terr := objectPath.Remove(ctx)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error removing file %s: %w\", objectPath, err)\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc (p *GSPath) RemoveAllVersions(ctx context.Context) error {\n\treturn p.Remove(ctx)\n}\n\nfunc (p *GSPath) Join(relativePath ...string) Path {\n\targs := []string{p.key}\n\targs = append(args, relativePath...)\n\tjoined := path.Join(args...)\n\treturn &GSPath{\n\t\tvfsContext: p.vfsContext,\n\t\tbucket:     p.bucket,\n\t\tkey:        joined,","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/util/pkg/vfs/gsfs.go#L137-L173","documentation":"GSPath.RemoveAll lists the whole object tree under the gs:// prefix with ReadTree, then calls Remove on each object. This error wraps a per-object Remove failure (which itself wraps the GCS Delete error) so the caller sees which object failed and why. Because removal aborts on the first failure, the store can be left partially deleted; a re-run is generally safe since the tree is re-listed.","triggerScenarios":"Calling GSPath.RemoveAll(ctx) (or cluster-destroy flows that empty a gs:// state store) when any listed object's Delete call fails: IAM denies storage.objects.delete, a 404 race after concurrent deletion, GCS transient errors exceeding the backoff budget for that object, or retention policy/holds on individual objects.","commonSituations":"kops cluster teardown against GCS state store with a service account that can list but not delete objects; two operators destroying the same cluster concurrently causing 404s mid-loop; bucket with retention configuration; transient Google API outage during a batch delete of many objects.","solutions":["Read the wrapped cause (googleapi Code): 403 -> grant the service account objectAdmin on the bucket; 404 -> object already deleted, safe to re-run; 412 -> clear retention/hold on the object.","Simply re-run the destroy/delete command — ReadTree re-lists, so previously deleted objects are skipped and the loop resumes where it failed.","Confirm IAM: `gsutil iam get gs://bucket` and ensure the identity used has roles/storage.objectAdmin; also check that the same identity is used by kops (GOOGLE_APPLICATION_CREDENTIALS / metadata server).","Avoid concurrent deletions of the same state store (lock operation or single operator) to eliminate 404 races."],"exampleFix":"// before\nfor _, objectPath := range tree {\n    if err := objectPath.Remove(ctx); err != nil {\n        return err // aborts whole teardown on one transient 404\n    }\n}\n// after\nfor _, objectPath := range tree {\n    if err := objectPath.Remove(ctx); err != nil {\n        var gerr *googleapi.Error\n        if errors.As(err, &gerr) && gerr.Code == 404 {\n            continue // concurrently deleted; keep going\n        }\n        return err\n    }\n}","handlingStrategy":"retry","validationCode":"// Pre-flight IAM + listing check before a bulk RemoveAll:\nfunc preflight(ctx context.Context, client *storage.Client, bucket, prefix string) error {\n    if _, err := client.Bucket(bucket).Attrs(ctx); err != nil {\n        return fmt.Errorf(\"cannot access bucket %s: %w\", bucket, err)\n    }\n    it := client.Bucket(bucket).Objects(ctx, &storage.Query{Prefix: prefix})\n    if _, err := it.Next(); err != nil && err != iterator.Done {\n        return fmt.Errorf(\"cannot list prefix %s: %w\", prefix, err)\n    }\n    return nil // listing works; 403 on list usually implies 403 on delete\n}","typeGuard":"func isNotFound(err error) bool {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) && gerr.Code == 404 { return true }\n    if st, ok := status.FromError(err); ok && st.Code() == codes.NotFound { return true }\n    return false\n}","tryCatchPattern":"err := gsPath.RemoveAll(ctx)\nif err != nil {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) && gerr.Code == 404 {\n        // concurrent deletion race - safe to re-run\n        err = gsPath.RemoveAll(ctx)\n    }\n}\nreturn err // remaining failures need IAM/retention fixes","preventionTips":["Verify service account IAM (roles/storage.objectAdmin) on the state bucket before cluster teardown.","Run bulk deletes single-threaded or make them idempotent; a 404 mid-run means another actor deleted the object.","Expect partial progress: RemoveAll aborts on first failure and is safe to re-run since the tree is re-listed.","Clear retention policies/holds on objects before deleting them."],"tags":["gcs","google-cloud","delete","kops"],"backgroundTag":"gcs-object-delete-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}