kubernetes/kops · error

found multiple kOps NatGateways in route table %s

Error message

found multiple kOps NatGateways in route table %s

What it means

After filtering the route table's multiple NAT gateways by cluster tag, more than one gateway remained tagged as belonging to this cluster. kOps cannot decide which one the route should point to and fails the lookup.

Source

Thrown at upup/pkg/fi/cloudup/awstasks/natgateway.go:245

				clusterName, ok := routeTable.Tags[awsup.TagClusterName]
				if !ok {
					return nil, fmt.Errorf("Could not find '%s' tag from route table", awsup.TagClusterName)
				}
				filteredNatGateways := []*ec2types.NatGateway{}
				for _, natGatewayID := range natGatewayIDs {
					gw, err := findNatGatewayById(ctx, cloud, fi.ValueOf(natGatewayID))
					if err != nil {
						return nil, err
					}

					if raws.HasOwnedTag(string(ec2types.ResourceTypeNatgateway)+":"+fi.ValueOf(natGatewayID), gw.Tags, clusterName) {
						filteredNatGateways = append(filteredNatGateways, gw)
					}
				}
				if len(filteredNatGateways) == 0 {
					klog.V(2).Infof("no kOps NatGateway found in route table %s", *rt.RouteTableId)
				} else if len(filteredNatGateways) > 1 {
					return nil, fmt.Errorf("found multiple kOps NatGateways in route table %s", *rt.RouteTableId)
				} else {
					return filteredNatGateways[0], nil
				}
			} else {
				return findNatGatewayById(ctx, cloud, fi.ValueOf(natGatewayIDs[0]))
			}
		}
	}

	return nil, nil
}

func (s *NatGateway) CheckChanges(a, e, changes *NatGateway) error {
	// New
	if a == nil {
		if !fi.ValueOf(e.Shared) {
			if e.ElasticIP == nil {
				return fi.RequiredField("ElasticIP")

View on GitHub (pinned to 4c8573c808)

Solutions

1. Delete the orphaned/extra NAT gateway: `aws ec2 delete-nat-gateway --nat-gateway-id nat-EXTRA` (keep the one the active route uses)
2. Remove the stale cluster tag from the gateway you intend to retire so filtering yields one
3. Prevent concurrent kops update runs on the same cluster (CI locking / single operator)
4. Re-run `kops update cluster --yes` after cleanup so routes converge to the single remaining gateway

Example fix

// before: two cluster-tagged NAT gateways on the route table
aws ec2 describe-nat-gateways --filter Name=tag:KubernetesCluster,Values=mycluster
// after: delete the orphan
aws ec2 delete-nat-gateway --nat-gateway-id nat-ORPHAN

out, _ := ec2Client.DescribeNatGateways(ctx, &ec2.DescribeNatGatewaysInput{
  Filter: []ec2types.Filter{{Name: aws.String("tag:KubernetesCluster"), Values: []string{clusterName}}}})
if len(out.NatGateways) > 1 { fmt.Println("multiple cluster NAT gateways — delete orphans before kops update") }

func singleClusterGateway(gws []ec2types.NatGateway) *ec2types.NatGateway {
  if len(gws) == 1 { return &gws[0] }
  return nil
}

ngw, err := findNatGatewayFromRouteTable(ctx, cloud, rt)
if err != nil && strings.Contains(err.Error(), "found multiple kOps NatGateways") {
  // enumerate gateways, delete the orphan, retry update
}

When it happens

Trigger: len(filteredNatGateways) > 1 — the route table has multiple nat- routes AND findNatGatewayById confirms each is a live, cluster-tagged (KubernetesCluster == this cluster) NAT gateway.

Common situations: A previous kops run left an orphaned NAT gateway still tagged for the cluster while a new one was also created; concurrent/overlapping `kops update cluster` runs; manual NAT gateway creation cloned tags from the original.

Related errors

• Could not find '%s' tag from route table
• found multiple Instances with name: %s
• found multiple tags for: %v
• error listing NatGateway %q: %v
• found multiple NatGateways with id %q

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/9b9190fdb61cb476. Report an issue: GitHub.