kubernetes/kops · error

error deleting %s: %w

Error message

error deleting %s: %w

What it means

GSPath.Remove deletes a single GCS object via client.Bucket(...).Object(...).Delete(ctx), wrapped in RetryWithBackoff. When the GCS API returns an error on every attempt (or a non-retryable error immediately), the error is wrapped as "error deleting <gs://path>", preserving the underlying googleapi error. The TODO in the source notes that a not-exists response is not yet translated to os.NotExist, so even a 'delete of an already-absent object' surfaces through this message.

Source

Thrown at util/pkg/vfs/gsfs.go:131

// Client returns the storage.Client bound to this path
func (p *GSPath) Client(ctx context.Context) (*storage.Client, error) {
	return p.getStorageClient(ctx)
}

func (p *GSPath) String() string {
	return p.Path()
}

func (p *GSPath) Remove(ctx context.Context) error {
	done, err := RetryWithBackoff(gcsWriteBackoff, func() (bool, error) {
		client, err := p.getStorageClient(ctx)
		if err != nil {
			return false, err
		}
		if err := client.Bucket(p.bucket).Object(p.key).Delete(ctx); err != nil {
			// TODO: Check for not-exists, return os.NotExist
			return false, fmt.Errorf("error deleting %s: %w", p, err)
		}

		return true, nil
	})
	if err != nil {
		return err
	} else if done {
		return nil
	} else {
		// Shouldn't happen - we always return a non-nil error with false
		return wait.ErrWaitTimeout
	}
}

func (p *GSPath) RemoveAll(ctx context.Context) error {
	tree, err := p.ReadTree(ctx)
	if err != nil {
		return err

View on GitHub (pinned to 4c8573c808)

Solutions

  1. Unwrap with errors.As into *googleapi.Error (or *apierror.APIError) and check Code: 404 means the object is already gone — treat as success; 403 means fix IAM (grant roles/storage.objectAdmin or storage.objects.delete on the bucket); 412 means a retention policy/hold is blocking deletion (remove the hold or wait out the retention period).
  2. Verify credentials: set GOOGLE_APPLICATION_CREDENTIALS to a service-account JSON with object delete rights, or fix gcloud application-default login; confirm with `gsutil rm gs://bucket/key` using the same identity.
  3. If transient (5xx / timeouts), simply retry the operation later — RemoveAll re-lists the tree, so already-deleted objects are skipped.
  4. Check bucket configuration for retentionPolicy, eventBasedHold, or temporaryHold via `gsutil retention ls gs://bucket` and clear holds if appropriate.

Example fix

// before
if err := p.Remove(ctx); err != nil {
    return err // 404 on already-deleted object treated as fatal
}
// after
if err := p.Remove(ctx); err != nil {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) && gerr.Code == 404 {
        return nil // object already absent
    }
    return err
}
Defensive patterns

Strategy: type-guard

Validate before calling

// Pre-flight: confirm credentials can delete an object in the bucket before batch deletes.
func canDeleteObject(ctx context.Context, client *storage.Client, bucket string) error {
    // Requires storage.buckets.get permission
    _, err := client.Bucket(bucket).Attrs(ctx)
    return err // 403 here almost certainly means Delete will fail with 403 too
}

Type guard

func isNotFound(err error) bool {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) && gerr.Code == 404 { return true }
    if st, ok := status.FromError(err); ok && st.Code() == codes.NotFound { return true }
    return false
}

func isGCSAuthError(err error) bool {
    var gerr *googleapi.Error
    return errors.As(err, &gerr) && gerr.Code == 403
}

Try / catch

if err := gsPath.Remove(ctx); err != nil {
    var gerr *googleapi.Error
    switch {
    case errors.As(err, &gerr) && gerr.Code == 404:
        // object already absent - idempotent success
    case errors.As(err, &gerr) && gerr.Code == 403:
        return fmt.Errorf("missing storage.objects.delete on bucket; grant roles/storage.objectAdmin: %w", err)
    case errors.As(err, &gerr) && gerr.Code == 412:
        return fmt.Errorf("object blocked by retention policy or hold: %w", err)
    default:
        return err // transient - safe to retry later
    }
}

Prevention

When it happens

Trigger: Calling GSPath.Remove(ctx) (directly or via RemoveAll, or via cluster state deletion) when: the object does not exist (GCS returns 404 NotFound), the credentials lack storage.objects.delete on the bucket, the object is subject to a retention policy / retention lock / object hold, or transient GCS errors persist past the backoff deadline.

Common situations: Destroying a kops cluster whose state lives in gs://<bucket> where the service account has read-only access (missing roles/storage.objectAdmin); bucket-level retention policies or litigation holds blocking object deletion; anonymous/none credentials because GOOGLE_APPLICATION_CREDENTIALS is unset or the metadata server is unreachable in the CI environment; deleting an already-cleaned-up store (404).

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/18af8a7cc35b98b6. Report an issue: GitHub.