kubernetes/kops · error

removing file %s: %w

Error message

removing file %s: %w

What it means

AzureBlobPath.RemoveAll lists all blobs under the path (the tree) and deletes each individually; if any single blob Remove fails, the operation aborts and wraps the error as `removing file %s: %w` with the blob path and underlying cause (403, 404 handling, lease conflicts, network).

Source

Thrown at util/pkg/vfs/azureblob.go:246

	if err != nil {
		return err
	}

	return nil
}

func (p *AzureBlobPath) RemoveAll(ctx context.Context) error {
	klog.V(8).Infof("Removing ALL files: %s", p)

	tree, err := p.ReadTree(ctx)
	if err != nil {
		return err
	}

	for _, blobPath := range tree {
		err := blobPath.Remove(ctx)
		if err != nil {
			return fmt.Errorf("removing file %s: %w", blobPath, err)
		}
	}

	return nil
}

func (p *AzureBlobPath) RemoveAllVersions(ctx context.Context) error {
	klog.V(8).Infof("Removing ALL file versions: %s", p)

	tree, err := p.ReadTree(ctx)
	if err != nil {
		return err
	}

	for _, blobPath := range tree {
		err := blobPath.Remove(ctx)
		if err != nil {
			return fmt.Errorf("removing file %s: %w", blobPath, err)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Regenerate credentials/SAS with delete (and list) permissions, e.g. account key or SAS with d+l flags.
2. Check for Azure soft-delete / immutability policies or active leases on the blobs and disable/allow delete.
3. Retry the operation — RemoveAll is resumable since already-deleted blobs are simply skipped on the next listing.
4. Verify the container exists and the account name/key in your Azure config are correct.

Example fix

// before: SAS without delete
sas := "?sv=...&sp=rl" // read+list only
// after
sas := "?sv=...&sp=rl d" // sp=rld : add delete permission

// Go: verify delete permission before RemoveAll
svc, _ := azblob.NewClient("https://acct.blob.core.windows.net/", cred)
_, err := svc.DeleteBlob(ctx, "container", "probe-blob", nil)
if err != nil {
    if azerr, ok := err.(*azcore.ResponseError); ok && azerr.StatusCode == 403 {
        return fmt.Errorf("credentials lack delete permission on container")
    }
}
// probe blob may not exist; 404 is fine — 403 means perms problem

func isAuthOrPolicyError(err error) bool {
    var re *azcore.ResponseError
    if errors.As(err, &re) {
        return re.StatusCode == 403 || re.StatusCode == 409
    }
    return false
}

err := path.RemoveAll(ctx)
if err != nil {
    if isAuthOrPolicyError(err) {
        // fix SAS/IAM or immutability policy, then retry
    }
    // transient: retry with backoff; RemoveAll resumes since deleted blobs vanish from listing
    retry.Do(func() error { return path.RemoveAll(ctx) }, retry.Attempts(3))
}

When it happens

Trigger: Calling RemoveAll on an azureblob VFSPath where one or more blobs under the prefix cannot be deleted — e.g. missing storage-account key or SAS lacking delete permission, blob immutable/leased, container missing, or transient API failure mid-deletion.

Common situations: `kops delete cluster` against an Azure state store with a read-only or deletion-restricted SAS token; soft-delete/immutability policies on the storage account blocking blob deletion; expired SAS token partway through a long tree delete.

Related errors

• getting VM: %w
• getting VMSS VM: %w
• DNS not implemented on azureCloud
• FindVPCInfo not implemented on azureCloud, use FindVNetInfo
• DeleteGroup not implemented on azureCloud

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/483aab4bb29f9b99. Report an issue: GitHub.