kubernetes/kops · error

error listing NatGateway %q: %v

Error message

error listing NatGateway %q: %v

What it means

DescribeNatGateways errored for the given NAT Gateway ID while resolving it by id (findNatGatewayById); the AWS API call itself failed — invalid/deleted ID, permissions, or throttling — as reported by the wrapped error.

Source

Thrown at upup/pkg/fi/cloudup/awstasks/natgateway.go:192

		}
		t := response.Tags[0]
		id = t.Value
		klog.V(2).Infof("Found NatGateway via subnet tag: %v", *id)
	}

	if id != nil {
		return findNatGatewayById(ctx, cloud, fi.ValueOf(id))
	}

	return nil, nil
}

func findNatGatewayById(ctx context.Context, cloud awsup.AWSCloud, id string) (*ec2types.NatGateway, error) {
	request := &ec2.DescribeNatGatewaysInput{}
	request.NatGatewayIds = []string{id}
	response, err := cloud.EC2().DescribeNatGateways(ctx, request)
	if err != nil {
		return nil, fmt.Errorf("error listing NatGateway %q: %v", id, err)
	}

	if response == nil || len(response.NatGateways) == 0 {
		klog.V(2).Infof("Unable to find NatGateway %q", id)
		return nil, nil
	}
	if len(response.NatGateways) != 1 {
		return nil, fmt.Errorf("found multiple NatGateways with id %q", id)
	}
	return &response.NatGateways[0], nil
}

func findNatGatewayFromRouteTable(ctx context.Context, cloud awsup.AWSCloud, routeTable *RouteTable) (*ec2types.NatGateway, error) {
	// Find via route on private route table
	if routeTable.ID != nil {
		klog.V(2).Infof("trying to match NatGateway via RouteTable %s", *routeTable.ID)
		rt, err := findRouteTableByID(ctx, cloud, *routeTable.ID)
		if err != nil {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped %v cause to identify the AWS error (NotFound vs AccessDenied vs throttling)
2. If InvalidNatGatewayID.NotFound, remove stale references (subnet AssociatedNatgateway tag / route table routes) and let kops recreate the gateway
3. Verify IAM permissions include ec2:DescribeNatGateways and ec2:CreateTags in the cluster region
4. Retry on throttling; reduce concurrent reconciliation or increase client rate limits

Example fix

// before: retrying blindly with a deleted gateway ID
findNatGatewayById(ctx, cloud, "nat-gone")
// after: clear the stale reference so kops provisions a new NAT gateway
aws ec2 delete-tags --resources subnet-0abc --tags Key=kops.k8s.io/AssociatedNatgateway
kops update cluster --name mycluster.example.com --yes

// verify API access and ID beforehand
_, err := ec2Client.DescribeNatGateways(ctx, &ec2.DescribeNatGatewaysInput{NatGatewayIds: []string{id}})
if err != nil { log.Printf("pre-check failed for %s: %v", id, err) }

func natGatewayExists(id string) bool {
  var aerr awserr.Error
  _ = aerr // inspect err code: InvalidNatGatewayID.NotFound → stale reference
  return false
}

gw, err := findNatGatewayById(ctx, cloud, id)
if err != nil {
  var nf *smithy.GenericAPIError
  if errors.As(err, &nf) && nf.Code == "InvalidNatGatewayID.NotFound" {
    // clear stale references and let kops recreate
  } else if isThrottling(err) {
    // exponential backoff retry
  }
}

When it happens

Trigger: cloud.EC2().DescribeNatGateways returns err — e.g. InvalidNatGatewayID.NotFound for a stale ID, AccessDenied on the EC2 API, request throttling, or connectivity failure.

Common situations: NAT gateway was deleted out-of-band so stored IDs are stale; IAM policy lacks ec2:DescribeNatGateways; API rate limits during large cluster reconciliation; regional outage.

Related errors

• describing instance for arn %q
• error terminating instances: %v
• error describing instances: %v
• error describing addresses: %v
• error describing volumes: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/6449bd57237465fe. Report an issue: GitHub.