kubernetes/kops · error

error applying DNS changeset for zone %s: %v

Error message

error applying DNS changeset for zone %s: %v

What it means

runOnce applies the accumulated changeset per zone via changeset.Apply(ctx). If a provider (Route53, CloudDNS, etc.) rejects the change batch, the error is logged as a warning, aggregated, and the first zone's error is returned wrapped as 'error applying DNS changeset for zone %s: %v'. It signals the provider refused an update batch for that hosted zone.

Source

Thrown at dns-controller/pkg/dns/dnscontroller.go:327

		newValues := newValueMap[k]
		if newValues == nil {
			err := op.deleteRecords(k)
			if err != nil {
				klog.Infof("error deleting records for %s: %v", k, err)
				errors = append(errors, err)
			}
		}
	}

	for key, changeset := range op.changesets {
		if changeset.IsEmpty() {
			continue
		}

		klog.V(2).Infof("Applying DNS changeset for zone %s", key)
		if err := changeset.Apply(ctx); err != nil {
			klog.Warningf("error applying DNS changeset for zone %s: %v", key, err)
			errors = append(errors, fmt.Errorf("error applying DNS changeset for zone %s: %v", key, err))
		}
	}

	if len(errors) != 0 {
		return errors[0]
	}

	// Success!  Store the snapshot as our new baseline
	c.mutex.Lock()
	defer c.mutex.Unlock()
	c.lastSuccessfulSnapshot = snapshot
	return nil
}

func (c *DNSController) RemoveRecordsImmediate(records []Record) error {
	ctx := context.TODO()

	op, err := newDNSOp(c.zoneRules, c.dnsCache)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped inner error: if throttled, the runWatcher loop will retry on the next tick; reduce update frequency or batch size.
2. Fix IAM permissions for the controller's identity on the specific hosted zones (route53:ChangeResourceRecordSets, list/get zones).
3. Verify the zone still exists and credentials are valid; re-check dns-provider flags and zone filtering (--zone/--zoneid).
4. For persistent payload errors, inspect the record values the controller derived from the service/ingress annotation.

Example fix

// before: broad IAM policy lacking route53 change rights
// after: attach to the controller's role
{
  "Effect": "Allow",
  "Action": ["route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets", "route53:ListHostedZones"],
  "Resource": ["arn:aws:route53:::hostedzone/YOURZONEID"]
}

// pre-flight: verify the zone is reachable and credentials work
zones, err := provider.Zones()
if err != nil {
    return fmt.Errorf("cannot list zones before applying changesets: %w", err)
}
_ = zones

if err := changeset.Apply(ctx); err != nil {
    var throttled bool
    if strings.Contains(err.Error(), "Throttling") {
        throttled = true
    }
    if throttled {
        time.Sleep(backoff) // then re-apply or wait for next tick
    } else {
        klog.Errorf("zone %s changeset rejected: %v", key, err)
    }
}

When it happens

Trigger: changeset.Apply(ctx) returns a provider error for a zone: rate limiting/throttling, invalid record payload, permission denied on the hosted zone, or transient API failure.

Common situations: AWS Route53 throttling (ThrottlingException) under many pods/services updating at once; IAM policy missing route53:ChangeResourceRecordSets; record TTL/value constraints; DNS API outage.

Related errors

• DNS ZoneID not set
• error deleting route53 record %q: %v
• error querying for route53 zones: %w
• error listing DNS ResourceRecords: %v
• error creating ResourceRecordSets: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/cd9fd0867ce5b24f. Report an issue: GitHub.