vitessio/vitess · error

unable to delete %q from bucket %q: %v

Error message

unable to delete %q from bucket %q: %v

What it means

RemoveBackup in GCS backup storage deletes each object under the backup's prefix; if the GCS Object.Delete call fails for an object, this error wraps the object name, bucket, and underlying GCS error. AbortBackup also routes through RemoveBackup, so aborted backups surface the same failure.

Source

Thrown at go/vt/mysqlctl/gcsbackupstorage/gcs.go:223

		return err
	}

	// Find all objects with the right prefix.
	query := &storage.Query{
		Prefix: objName(dir, name, "" /* include trailing slash */),
	}
	// Delete all the found objects.
	it := c.Bucket(bucket).Objects(ctx, query)
	for {
		obj, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			return err
		}
		if err := c.Bucket(bucket).Object(obj.Name).Delete(ctx); err != nil {
			return fmt.Errorf("unable to delete %q from bucket %q: %v", obj.Name, bucket, err)
		}
	}
	return nil
}

// Close implements BackupStorage.
func (bs *GCSBackupStorage) Close() error {
	bs.mu.Lock()
	defer bs.mu.Unlock()

	if bs._client != nil {
		// If client.Close() fails, we still clear bs._client,
		// so we know to create a new client the next time one
		// is needed.
		client := bs._client
		defer func() {
			bs._client = nil
		}()

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Grant storage.objects.delete (or storage.admin) on the bucket to the service account.
2. Retry the removal — GCS errors are often transient; use the returned error to distinguish permanent (403) from transient (5xx/429).
3. If the object was already deleted, verify the backup listing and clean up leftover manifest references manually.

Example fix

// before
// default IAM: roles/storage.objectViewer only
// after
gsutil iam ch serviceAccount:sa@proj.iam.gserviceaccount.com:roles/storage.objectAdmin gs://my-bucket

// pre-check IAM before delete flow
resp, err := c.Bucket(bucket).IAM().TestPermissions(ctx, []string{"storage.objects.delete"})
if err != nil || len(resp) == 0 { return errors.New("missing storage.objects.delete permission") }

err := fbs.RemoveBackup(ctx, dir, name)
if err != nil {
	var pre *googleapi.Error
	if errors.As(err, &pre) && pre.Code == 403 { /* permanent: fix IAM */ }
	// otherwise retry with backoff
}

When it happens

Trigger: Deleting a backup whose object Delete returns a GCS API error: permission denied, object already gone with non-retryable condition, bucket reauth failure, or quota/rate issue.

Common situations: IAM principal lacking storage.objects.delete permission; backup partially deleted by a prior run leaving transient GCS errors; expired/invalid service-account credentials; GCS outage or rate limiting during AbortBackup.

Related errors

• AddFile cannot be called on read-only backup
• EndBackup cannot be called on read-only backup
• AbortBackup cannot be called on read-only backup
• ReadFile cannot be called on read-write backup
• ReadFile cannot be called on read-write backup

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/0d4340b551aa0df6. Report an issue: GitHub.