{"record":{"id":"0d4340b551aa0df6","repo":"vitessio/vitess","slug":"unable-to-delete-q-from-bucket-q-v","errorCode":null,"errorMessage":"unable to delete %q from bucket %q: %v","messagePattern":"unable to delete %q from bucket %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/mysqlctl/gcsbackupstorage/gcs.go","lineNumber":223,"sourceCode":"\t\treturn err\n\t}\n\n\t// Find all objects with the right prefix.\n\tquery := &storage.Query{\n\t\tPrefix: objName(dir, name, \"\" /* include trailing slash */),\n\t}\n\t// Delete all the found objects.\n\tit := c.Bucket(bucket).Objects(ctx, query)\n\tfor {\n\t\tobj, err := it.Next()\n\t\tif err == iterator.Done {\n\t\t\tbreak\n\t\t}\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tif err := c.Bucket(bucket).Object(obj.Name).Delete(ctx); err != nil {\n\t\t\treturn fmt.Errorf(\"unable to delete %q from bucket %q: %v\", obj.Name, bucket, err)\n\t\t}\n\t}\n\treturn nil\n}\n\n// Close implements BackupStorage.\nfunc (bs *GCSBackupStorage) Close() error {\n\tbs.mu.Lock()\n\tdefer bs.mu.Unlock()\n\n\tif bs._client != nil {\n\t\t// If client.Close() fails, we still clear bs._client,\n\t\t// so we know to create a new client the next time one\n\t\t// is needed.\n\t\tclient := bs._client\n\t\tdefer func() {\n\t\t\tbs._client = nil\n\t\t}()","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/mysqlctl/gcsbackupstorage/gcs.go#L205-L241","documentation":"RemoveBackup in GCS backup storage deletes each object under the backup's prefix; if the GCS Object.Delete call fails for an object, this error wraps the object name, bucket, and underlying GCS error. AbortBackup also routes through RemoveBackup, so aborted backups surface the same failure.","triggerScenarios":"Deleting a backup whose object Delete returns a GCS API error: permission denied, object already gone with non-retryable condition, bucket reauth failure, or quota/rate issue.","commonSituations":"IAM principal lacking storage.objects.delete permission; backup partially deleted by a prior run leaving transient GCS errors; expired/invalid service-account credentials; GCS outage or rate limiting during AbortBackup.","solutions":["Grant storage.objects.delete (or storage.admin) on the bucket to the service account.","Retry the removal — GCS errors are often transient; use the returned error to distinguish permanent (403) from transient (5xx/429).","If the object was already deleted, verify the backup listing and clean up leftover manifest references manually."],"exampleFix":"// before\n// default IAM: roles/storage.objectViewer only\n// after\ngsutil iam ch serviceAccount:sa@proj.iam.gserviceaccount.com:roles/storage.objectAdmin gs://my-bucket","handlingStrategy":"retry","validationCode":"// pre-check IAM before delete flow\nresp, err := c.Bucket(bucket).IAM().TestPermissions(ctx, []string{\"storage.objects.delete\"})\nif err != nil || len(resp) == 0 { return errors.New(\"missing storage.objects.delete permission\") }","typeGuard":null,"tryCatchPattern":"err := fbs.RemoveBackup(ctx, dir, name)\nif err != nil {\n\tvar pre *googleapi.Error\n\tif errors.As(err, &pre) && pre.Code == 403 { /* permanent: fix IAM */ }\n\t// otherwise retry with backoff\n}","preventionTips":["Grant storage.objectAdmin to the backup service account","Retry transient GCS 5xx/429 errors with backoff","Monitor AbortBackup failures and alert on repeated delete errors"],"tags":["gcs","backup-storage","permissions"],"backgroundTag":"gcs-delete-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}