What causes "error querying resource records for zone %q: %v" in kubernetes/kops?

Inspect the wrapped provider error: throttle/transient errors resolve on the next watcher tick; add backoff or widen the loop interval.

kubernetes/kops · error

error querying resource records for zone %q: %v

Error message

error querying resource records for zone %q: %v

What it means

After confirming record-set support, listRecords calls rrsProvider.List() to fetch all records in the zone. A provider error there is wrapped as 'error querying resource records for zone %q: %v'. The result is cached per zone (name::id), so a failure forces re-query on every pass until it succeeds.

Source

Thrown at dns-controller/pkg/dns/dnscontroller.go:487

	return changeset, nil
}

// listRecords is a wrapper around listing records, but will cache the results for the duration of the dnsOp
func (o *dnsOp) listRecords(zone dnsprovider.Zone) ([]dnsprovider.ResourceRecordSet, error) {
	key := zone.Name() + "::" + zone.ID()

	rrs := o.recordsCache[key]
	if rrs == nil {
		rrsProvider, ok := zone.ResourceRecordSets()
		if !ok {
			return nil, fmt.Errorf("zone does not support resource records %q", zone.Name())
		}

		klog.V(2).Infof("Querying all dnsprovider records for zone %q", zone.Name())
		var err error
		rrs, err = rrsProvider.List()
		if err != nil {
			return nil, fmt.Errorf("error querying resource records for zone %q: %v", zone.Name(), err)
		}

		o.recordsCache[key] = rrs
	}

	return rrs, nil
}

func (o *dnsOp) deleteRecords(k recordKey) error {
	klog.V(2).Infof("Deleting all records for %s", k)

	fqdn := EnsureDotSuffix(k.FQDN)

	zone := o.findZone(fqdn)
	if zone == nil {
		// TODO: Post event into service / pod
		return fmt.Errorf("no suitable zone found for %q", fqdn)
	}

View on GitHub (pinned to 4c8573c808)

Solutions

Inspect the wrapped provider error: throttle/transient errors resolve on the next watcher tick; add backoff or widen the loop interval.
Grant record-listing permissions (route53:ListResourceRecordSets) to the controller identity.
Check network egress/proxy settings from the dns-controller pod to the provider endpoint.
For very large zones, consider scoping the controller with --zone/--zoneid filters to reduce list volume.

Example fix

// before: role can change records but not list them
//   AccessDenied: route53:ListResourceRecordSets
// after: add to the controller policy
{
  "Effect": "Allow",
  "Action": ["route53:ListResourceRecordSets"],
  "Resource": ["arn:aws:route53:::hostedzone/YOURZONEID"]
}

Defensive patterns

Strategy: retry

Validate before calling

// pre-flight IAM check (AWS example)
// aws route53 list-resource-record-sets --hosted-zone-id ZONEID --max-items 1
// run from the controller's identity before rollout

Try / catch

rrs, err := o.listRecords(zone)
if err != nil {
    if strings.Contains(err.Error(), "Throttling") || strings.Contains(err.Error(), "timeout") {
        time.Sleep(backoff) // transient: retry next tick
    }
    return err
}

Prevention

Grant route53:ListResourceRecordSets to the controller identity.
Widen the watcher interval when managing many/large zones to avoid throttling.
Monitor provider API error rates from the controller logs.
Scope with --zone/--zoneid to limit zones listed per pass.

When it happens

Trigger: rrsProvider.List() errors: API throttling, network timeout to the DNS API, permission denied on record listing, malformed pagination in the provider driver.

Common situations: Route53 ThrottlingException with many zones/pods; IAM missing route53:ListResourceRecordSets; transient network outage from the cluster to the DNS API; very large hosted zones hitting driver limits.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/178bb78147739486. Report an issue: GitHub.