deregistering cloud instance %s of group %q: removing IP fro

Error message

deregistering cloud instance %s of group %q: removing IP from lb: %w

What it means

When the instance's IP is found in a back-end pool, DeregisterInstance calls lbAPI.RemoveBackendServers to detach it; errors are wrapped as "deregistering cloud instance %s of group %q: removing IP from lb: %w". The IP and back-end were identified correctly but the mutation against the Scaleway LB API failed, so the node may still receive traffic.

Source

Thrown at upup/pkg/fi/cloudup/scaleway/cloud.go:289

	}
	for _, loadBalancer := range lbs {
		backEnds, err := s.lbAPI.ListBackends(&lb.ZonedAPIListBackendsRequest{
			Zone: s.zone,
			LBID: loadBalancer.ID,
		}, scw.WithAllPages())
		if err != nil {
			return fmt.Errorf("deregistering cloud instance %s of group %q: listing load-balancer's back-ends for instance creation: %w", i.ID, i.CloudInstanceGroup.HumanName, err)
		}
		for _, backEnd := range backEnds.Backends {
			for _, ip := range backEnd.Pool {
				if ip == serverIP {
					_, err := s.lbAPI.RemoveBackendServers(&lb.ZonedAPIRemoveBackendServersRequest{
						Zone:      s.zone,
						BackendID: backEnd.ID,
						ServerIP:  []string{serverIP},
					})
					if err != nil {
						return fmt.Errorf("deregistering cloud instance %s of group %q: removing IP from lb: %w", i.ID, i.CloudInstanceGroup.HumanName, err)
					}
				}
			}
		}
	}

	return nil
}

func (s *scwCloudImplementation) DetachInstance(i *cloudinstances.CloudInstance) error {
	klog.V(8).Infof("Scaleway DetachInstance is not implemented yet")
	return fmt.Errorf("DetachInstance is not implemented yet for Scaleway")
}

// FindClusterStatus was used before etcd-manager to check the etcd cluster status and prevent unsupported changes.
func (s *scwCloudImplementation) FindClusterStatus(cluster *kops.Cluster) (*kops.ClusterStatus, error) {
	klog.V(8).Info("Scaleway FindClusterStatus is not implemented")
	return nil, nil

View on GitHub (pinned to 4c8573c808)

Solutions

Check the wrapped status code: 409/404 usually means the IP was already removed or backend changed — verify the pool in the console and treat as done.
Ensure s.zone matches the LB/backend zone.
Grant the IAM policy lb:update (back-end mutation) permission.
Retry after the LB finishes any in-progress operation or transient error clears.

Example fix

// before
// concurrent deregister: second RemoveBackendServers 404s
// after: check pool membership first / tolerate not-found
// if !backend.Pool contains serverIP { skip RemoveBackendServers }

Defensive patterns

Strategy: try-catch

Try / catch

_, err := s.lbAPI.RemoveBackendServers(&lb.ZonedAPIRemoveBackendServersRequest{Zone: s.zone, BackendID: backEnd.ID, ServerIP: []string{serverIP}})
if err != nil {
  var respErr *scw.ResponseError
  if errors.As(err, &respErr) && (respErr.StatusCode == 404 || respErr.StatusCode == 409) {
    klog.Infof("IP %s already removed from backend %s", serverIP, backEnd.ID)
    return nil
  }
  return fmt.Errorf("removing IP from lb backend %s: %w", backEnd.ID, err)
}

Prevention

Serialize rolling updates so only one controller deregisters a node at a time.
Treat 404/409 on RemoveBackendServers as success (idempotent deregistration).
Grant lb:update permission to the kops IAM key.
Wait for LB configuration-drain windows before retrying failed removals.

When it happens

Trigger: lbAPI.RemoveBackendServers(&lb.ZonedAPIRemoveBackendServersRequest{Zone: s.zone, BackendID: backEnd.ID, ServerIP: []string{serverIP}}) fails: backend ID stale/deleted concurrently, cross-zone request, IP no longer in pool (conflict), LB under another operation (draining/locking), permission denied, or transient API error.

Common situations: Two rolling-update controllers deregistering the same node concurrently; LB in a transitional state right after a configuration change; IAM key lacking lb:update permission; zone mismatch on a zoned LB.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/b2ff9cbb519e4c4d. Report an issue: GitHub.