kubernetes/kops · error

loadbalancer has gone into ERROR state

Error message

loadbalancer has gone into ERROR state

What it means

waitLoadbalancerActiveProvisioningStatus polls Octavia GET /lbaas/loadbalancers/{id} with exponential backoff (~5 min, 22 steps). If the load balancer's provisioning_status becomes ERROR, the wait returns this error immediately instead of continuing to poll. It signals Octavia gave up provisioning the LB (amphora failed to spawn, bad subnet, quota, unsupported flavor) rather than it merely being slow.

Source

Thrown at upup/pkg/fi/cloudup/openstacktasks/lb.go:79

func waitLoadbalancerActiveProvisioningStatus(client *gophercloud.ServiceClient, loadbalancerID string) (string, error) {
	backoff := wait.Backoff{
		Duration: loadbalancerActiveInitDelay,
		Factor:   loadbalancerActiveFactor,
		Steps:    loadbalancerActiveSteps,
	}

	var provisioningStatus string
	err := wait.ExponentialBackoff(backoff, func() (bool, error) {
		loadbalancer, err := loadbalancers.Get(context.TODO(), client, loadbalancerID).Extract()
		if err != nil {
			return false, err
		}
		provisioningStatus = loadbalancer.ProvisioningStatus
		switch loadbalancer.ProvisioningStatus {
		case activeStatus:
			return true, nil
		case errorStatus:
			return true, fmt.Errorf("loadbalancer has gone into ERROR state")
		default:
			klog.Infof("Waiting for Loadbalancer to be ACTIVE...")
			return false, nil
		}
	})

	if err == wait.ErrWaitTimeout {
		err = fmt.Errorf("loadbalancer failed to go into ACTIVE provisioning status within allotted time")
	}
	return provisioningStatus, err
}

// GetDependencies returns the dependencies of the Instance task
func (e *LB) GetDependencies(tasks map[string]fi.CloudupTask) []fi.CloudupTask {
	var deps []fi.CloudupTask
	for _, task := range tasks {
		if _, ok := task.(*Subnet); ok {
			deps = append(deps, task)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Inspect the LB and its listeners/pools: `openstack loadbalancer status show <id>` and check Octavia/amphora logs (`openstack loadbalancer amphora list`, amphora console)
2. Delete the ERROR-state loadbalancer and rerun `kops update cluster` to recreate it: `openstack loadbalancer delete <id>`
3. Verify the VIP subnet has free IPs and the amphora management network is reachable
4. Check Octavia service health/version compatibility (amphora image vs controller) with the cloud admin
5. If using a non-standard provider/flavor (e.g. ovn), confirm the cluster spec's LB provider and flavorID are supported

Example fix

# before: stuck ERROR LB from failed amphora
# openstack loadbalancer show <lb-id>   # provisioning_status: ERROR
openstack loadbalancer delete <lb-id>
kops update cluster --name <cluster> --yes
# after: new LB reaches ACTIVE provisioning_status

// pre-flight: check Octavia service availability and no pre-existing ERROR LB
func validateOctaviaReady(lbClient *gophercloud.ServiceClient, projectID string) error {
	allPages, err := loadbalancers.List(lbClient, loadbalancers.ListOpts{ProjectID: projectID}).AllPages()
	if err != nil {
		return fmt.Errorf("octavia API unreachable: %w", err)
	}
	lbs, _ := loadbalancers.ExtractLoadBalancers(allPages)
	for _, lb := range lbs {
		if lb.ProvisioningStatus == "ERROR" {
			return fmt.Errorf("loadbalancer %s already in ERROR state; delete before applying", lb.ID)
		}
	}
	return nil
}

func isLBErrorState(err error) bool {
	return strings.Contains(err.Error(), "gone into ERROR state")
}

status, err := waitLoadbalancerActiveProvisioningStatus(client, lbID)
if isLBErrorState(err) {
	// inspect/delete the ERROR LB, then recreate; plain retry cannot recover
	return fmt.Errorf("LB %s entered ERROR state; check amphora logs and `openstack loadbalancer delete %s` before re-applying: %w", lbID, lbID, err)
}
if errors.Is(err, wait.ErrWaitTimeout) {
	return fmt.Errorf("LB still not ACTIVE after ~5min; check octavia health")
}

When it happens

Trigger: During `kops update cluster` LB creation, the Octavia loadbalancer transitions to provisioning_status=ERROR: amphora instance failed to boot, VIP subnet/port invalid or full, provider flavor unsupported, certificate or networking issue on the amphora, or the backend provider (e.g. ovn) rejected the config.

Common situations: Octavia control plane unhealthy (amphora image/version mismatch after Octavia upgrade); exhausted floating-IP or subnet IP space for the VIP; using OVN provider with unsupported LB features; control-plane node flavor too small for the amphora; cloud outage during cluster create.

Related errors

• failed to build load balancer client: %w
• error building lb client: %w
• loadbalancer API versions not found
• GetApiIngressStatus: Failed to list openstack loadbalancers:
• error deleting loadbalancer: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/a1c0b2e40d42aa29. Report an issue: GitHub.