kubernetes/kops · error

error removing file %s: %w

Error message

error removing file %s: %w

What it means

GSPath.RemoveAll lists the whole object tree under the gs:// prefix with ReadTree, then calls Remove on each object. This error wraps a per-object Remove failure (which itself wraps the GCS Delete error) so the caller sees which object failed and why. Because removal aborts on the first failure, the store can be left partially deleted; a re-run is generally safe since the tree is re-listed.

Source

Thrown at util/pkg/vfs/gsfs.go:155

		return err
	} else if done {
		return nil
	} else {
		// Shouldn't happen - we always return a non-nil error with false
		return wait.ErrWaitTimeout
	}
}

func (p *GSPath) RemoveAll(ctx context.Context) error {
	tree, err := p.ReadTree(ctx)
	if err != nil {
		return err
	}

	for _, objectPath := range tree {
		err := objectPath.Remove(ctx)
		if err != nil {
			return fmt.Errorf("error removing file %s: %w", objectPath, err)
		}
	}

	return nil
}

func (p *GSPath) RemoveAllVersions(ctx context.Context) error {
	return p.Remove(ctx)
}

func (p *GSPath) Join(relativePath ...string) Path {
	args := []string{p.key}
	args = append(args, relativePath...)
	joined := path.Join(args...)
	return &GSPath{
		vfsContext: p.vfsContext,
		bucket:     p.bucket,
		key:        joined,

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped cause (googleapi Code): 403 -> grant the service account objectAdmin on the bucket; 404 -> object already deleted, safe to re-run; 412 -> clear retention/hold on the object.
2. Simply re-run the destroy/delete command — ReadTree re-lists, so previously deleted objects are skipped and the loop resumes where it failed.
3. Confirm IAM: `gsutil iam get gs://bucket` and ensure the identity used has roles/storage.objectAdmin; also check that the same identity is used by kops (GOOGLE_APPLICATION_CREDENTIALS / metadata server).
4. Avoid concurrent deletions of the same state store (lock operation or single operator) to eliminate 404 races.

Example fix

// before
for _, objectPath := range tree {
    if err := objectPath.Remove(ctx); err != nil {
        return err // aborts whole teardown on one transient 404
    }
}
// after
for _, objectPath := range tree {
    if err := objectPath.Remove(ctx); err != nil {
        var gerr *googleapi.Error
        if errors.As(err, &gerr) && gerr.Code == 404 {
            continue // concurrently deleted; keep going
        }
        return err
    }
}

// Pre-flight IAM + listing check before a bulk RemoveAll:
func preflight(ctx context.Context, client *storage.Client, bucket, prefix string) error {
    if _, err := client.Bucket(bucket).Attrs(ctx); err != nil {
        return fmt.Errorf("cannot access bucket %s: %w", bucket, err)
    }
    it := client.Bucket(bucket).Objects(ctx, &storage.Query{Prefix: prefix})
    if _, err := it.Next(); err != nil && err != iterator.Done {
        return fmt.Errorf("cannot list prefix %s: %w", prefix, err)
    }
    return nil // listing works; 403 on list usually implies 403 on delete
}

func isNotFound(err error) bool {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) && gerr.Code == 404 { return true }
    if st, ok := status.FromError(err); ok && st.Code() == codes.NotFound { return true }
    return false
}

err := gsPath.RemoveAll(ctx)
if err != nil {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) && gerr.Code == 404 {
        // concurrent deletion race - safe to re-run
        err = gsPath.RemoveAll(ctx)
    }
}
return err // remaining failures need IAM/retention fixes

When it happens

Trigger: Calling GSPath.RemoveAll(ctx) (or cluster-destroy flows that empty a gs:// state store) when any listed object's Delete call fails: IAM denies storage.objects.delete, a 404 race after concurrent deletion, GCS transient errors exceeding the backoff budget for that object, or retention policy/holds on individual objects.

Common situations: kops cluster teardown against GCS state store with a service account that can list but not delete objects; two operators destroying the same cluster concurrently causing 404s mid-loop; bucket with retention configuration; transient Google API outage during a batch delete of many objects.

Related errors

• error deleting %s: %w
• reading from stdin: %v
• reading file %q: %v
• building VFS path for %q: %w
• delete on clusters on %q not (yet) supported

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/88e9fd8c220b477e. Report an issue: GitHub.