kubernetes/kops · warning

got retryable error while deleting tags on %q, but retried t

Error message

got retryable error while deleting tags on %q, but retried too many times without success: %v

What it means

kOps wraps DeleteTags failures when AWS keeps returning a retryable 'eventual consistency' error (e.g. resource not yet visible to the tagging API) after DeleteTagsMaxAttempts retries. It signals that tag deletion was attempted repeatedly but the AWS side still reported the resource/tag as missing or not found.

Source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:1321

	ec2Tags := []ec2types.Tag{}
	for k, v := range tags {
		ec2Tags = append(ec2Tags, ec2types.Tag{Key: aws.String(k), Value: aws.String(v)})
	}

	attempt := 0
	for {
		attempt++

		request := &ec2.DeleteTagsInput{
			Tags:      ec2Tags,
			Resources: []string{resourceID},
		}

		_, err := c.EC2().DeleteTags(ctx, request)
		if err != nil {
			if isTagsEventualConsistencyError(err) {
				if attempt > DeleteTagsMaxAttempts {
					return fmt.Errorf("got retryable error while deleting tags on %q, but retried too many times without success: %v", resourceID, err)
				}

				if (attempt % DeleteTagsLogInterval) == 0 {
					klog.Infof("waiting for eventual consistency while deleting tags on %q", resourceID)
				}

				klog.V(2).Infof("will retry after encountering error deleting tags on %q: %v", resourceID, err)
				time.Sleep(DeleteTagsRetryInterval)
				continue
			}

			return fmt.Errorf("error deleting tags on %v: %v", resourceID, err)
		}

		return nil
	}
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Wait a few minutes and re-run the kops command (the resource state will have converged).
2. Use the latest kops release; DeleteTagsMaxAttempts/DeleteTagsRetryInterval tuning has improved over versions.
3. If the resource truly exists, verify the resource ID is correct with `aws ec2 describe-tags --filters Name=resource-id,Values=<id>`.
4. As a last resort, delete the tags manually with the AWS CLI and retry kops.

Example fix

// before: repeatedly hitting eventual consistency on a fresh resource
err := cloud.DeleteTags(id, tags) // retried too many times
// after: add delay/backoff before deleting tags on a newly created resource
time.Sleep(30 * time.Second) // or poll until tags visible via DescribeTags
err := cloud.DeleteTags(id, tags)

// Ensure the resource exists and tags are visible before deleting
out, err := ec2Client.DescribeTags(ctx, &ec2.DescribeTagsInput{Filters: []ec2types.Filter{{Name: aws.String("resource-id"), Values: []string{resourceID}}}})
if err != nil || len(out.Tags) == 0 { /* defer deletion or abort */ }

err := cloud.DeleteTags(ctx, resourceID, tagKeys)
var retriable *exhaustedRetriesError
if errors.As(err, &retriable) {
    klog.Warningf("tag deletion still eventually-inconsistent for %s; re-run later", resourceID)
}

When it happens

Trigger: Deleting tags on an EC2 resource (VPC, subnet, IGW, instance, etc.) immediately after resource creation or an immediate delete/recreate cycle, where the DescribeTags/DeleteTags APIs lag behind CreateTags.

Common situations: Rapidly reusing a VPC name right after deleting a cluster; tags deleted during teardown while AWS still returns InvalidResourceId/NotFound due to propagation delay; kops destroy-cluster racing AWS consistency.

Related errors

• found multiple Volumes with name: %s
• error adding AWS Tags to EBS Volume: %v
• found multiple EgressOnlyInternetGateways matching tags
• error querying tags for ElasticIP: %v
• Unable to tag subnet %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/4e0ba5fe1fda4462. Report an issue: GitHub.