{"record":{"id":"18af8a7cc35b98b6","repo":"kubernetes/kops","slug":"error-deleting-s-w","errorCode":null,"errorMessage":"error deleting %s: %w","messagePattern":"error deleting (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/pkg/vfs/gsfs.go","lineNumber":131,"sourceCode":"\n// Client returns the storage.Client bound to this path\nfunc (p *GSPath) Client(ctx context.Context) (*storage.Client, error) {\n\treturn p.getStorageClient(ctx)\n}\n\nfunc (p *GSPath) String() string {\n\treturn p.Path()\n}\n\nfunc (p *GSPath) Remove(ctx context.Context) error {\n\tdone, err := RetryWithBackoff(gcsWriteBackoff, func() (bool, error) {\n\t\tclient, err := p.getStorageClient(ctx)\n\t\tif err != nil {\n\t\t\treturn false, err\n\t\t}\n\t\tif err := client.Bucket(p.bucket).Object(p.key).Delete(ctx); err != nil {\n\t\t\t// TODO: Check for not-exists, return os.NotExist\n\t\t\treturn false, fmt.Errorf(\"error deleting %s: %w\", p, err)\n\t\t}\n\n\t\treturn true, nil\n\t})\n\tif err != nil {\n\t\treturn err\n\t} else if done {\n\t\treturn nil\n\t} else {\n\t\t// Shouldn't happen - we always return a non-nil error with false\n\t\treturn wait.ErrWaitTimeout\n\t}\n}\n\nfunc (p *GSPath) RemoveAll(ctx context.Context) error {\n\ttree, err := p.ReadTree(ctx)\n\tif err != nil {\n\t\treturn err","sourceCodeStart":113,"sourceCodeEnd":149,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/util/pkg/vfs/gsfs.go#L113-L149","documentation":"GSPath.Remove deletes a single GCS object via client.Bucket(...).Object(...).Delete(ctx), wrapped in RetryWithBackoff. When the GCS API returns an error on every attempt (or a non-retryable error immediately), the error is wrapped as \"error deleting <gs://path>\", preserving the underlying googleapi error. The TODO in the source notes that a not-exists response is not yet translated to os.NotExist, so even a 'delete of an already-absent object' surfaces through this message.","triggerScenarios":"Calling GSPath.Remove(ctx) (directly or via RemoveAll, or via cluster state deletion) when: the object does not exist (GCS returns 404 NotFound), the credentials lack storage.objects.delete on the bucket, the object is subject to a retention policy / retention lock / object hold, or transient GCS errors persist past the backoff deadline.","commonSituations":"Destroying a kops cluster whose state lives in gs://<bucket> where the service account has read-only access (missing roles/storage.objectAdmin); bucket-level retention policies or litigation holds blocking object deletion; anonymous/none credentials because GOOGLE_APPLICATION_CREDENTIALS is unset or the metadata server is unreachable in the CI environment; deleting an already-cleaned-up store (404).","solutions":["Unwrap with errors.As into *googleapi.Error (or *apierror.APIError) and check Code: 404 means the object is already gone — treat as success; 403 means fix IAM (grant roles/storage.objectAdmin or storage.objects.delete on the bucket); 412 means a retention policy/hold is blocking deletion (remove the hold or wait out the retention period).","Verify credentials: set GOOGLE_APPLICATION_CREDENTIALS to a service-account JSON with object delete rights, or fix gcloud application-default login; confirm with `gsutil rm gs://bucket/key` using the same identity.","If transient (5xx / timeouts), simply retry the operation later — RemoveAll re-lists the tree, so already-deleted objects are skipped.","Check bucket configuration for retentionPolicy, eventBasedHold, or temporaryHold via `gsutil retention ls gs://bucket` and clear holds if appropriate."],"exampleFix":"// before\nif err := p.Remove(ctx); err != nil {\n    return err // 404 on already-deleted object treated as fatal\n}\n// after\nif err := p.Remove(ctx); err != nil {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) && gerr.Code == 404 {\n        return nil // object already absent\n    }\n    return err\n}","handlingStrategy":"type-guard","validationCode":"// Pre-flight: confirm credentials can delete an object in the bucket before batch deletes.\nfunc canDeleteObject(ctx context.Context, client *storage.Client, bucket string) error {\n    // Requires storage.buckets.get permission\n    _, err := client.Bucket(bucket).Attrs(ctx)\n    return err // 403 here almost certainly means Delete will fail with 403 too\n}","typeGuard":"func isNotFound(err error) bool {\n    var gerr *googleapi.Error\n    if errors.As(err, &gerr) && gerr.Code == 404 { return true }\n    if st, ok := status.FromError(err); ok && st.Code() == codes.NotFound { return true }\n    return false\n}\n\nfunc isGCSAuthError(err error) bool {\n    var gerr *googleapi.Error\n    return errors.As(err, &gerr) && gerr.Code == 403\n}","tryCatchPattern":"if err := gsPath.Remove(ctx); err != nil {\n    var gerr *googleapi.Error\n    switch {\n    case errors.As(err, &gerr) && gerr.Code == 404:\n        // object already absent - idempotent success\n    case errors.As(err, &gerr) && gerr.Code == 403:\n        return fmt.Errorf(\"missing storage.objects.delete on bucket; grant roles/storage.objectAdmin: %w\", err)\n    case errors.As(err, &gerr) && gerr.Code == 412:\n        return fmt.Errorf(\"object blocked by retention policy or hold: %w\", err)\n    default:\n        return err // transient - safe to retry later\n    }\n}","preventionTips":["Grant the service account roles/storage.objectAdmin on the state bucket before any delete/destroy operation.","Ensure GOOGLE_APPLICATION_CREDENTIALS (or workload identity / metadata server) is set in CI environments.","Check bucket retention policies and holds before teardown: gsutil retention ls gs://bucket.","Make delete flows idempotent: treat 404 as success."],"tags":["gcs","google-cloud","delete","permissions","kops"],"backgroundTag":"gcs-object-delete-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}