{"record":{"id":"a1c0b2e40d42aa29","repo":"kubernetes/kops","slug":"loadbalancer-has-gone-into-error-state","errorCode":null,"errorMessage":"loadbalancer has gone into ERROR state","messagePattern":"loadbalancer has gone into ERROR state","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/cloudup/openstacktasks/lb.go","lineNumber":79,"sourceCode":"func waitLoadbalancerActiveProvisioningStatus(client *gophercloud.ServiceClient, loadbalancerID string) (string, error) {\n\tbackoff := wait.Backoff{\n\t\tDuration: loadbalancerActiveInitDelay,\n\t\tFactor:   loadbalancerActiveFactor,\n\t\tSteps:    loadbalancerActiveSteps,\n\t}\n\n\tvar provisioningStatus string\n\terr := wait.ExponentialBackoff(backoff, func() (bool, error) {\n\t\tloadbalancer, err := loadbalancers.Get(context.TODO(), client, loadbalancerID).Extract()\n\t\tif err != nil {\n\t\t\treturn false, err\n\t\t}\n\t\tprovisioningStatus = loadbalancer.ProvisioningStatus\n\t\tswitch loadbalancer.ProvisioningStatus {\n\t\tcase activeStatus:\n\t\t\treturn true, nil\n\t\tcase errorStatus:\n\t\t\treturn true, fmt.Errorf(\"loadbalancer has gone into ERROR state\")\n\t\tdefault:\n\t\t\tklog.Infof(\"Waiting for Loadbalancer to be ACTIVE...\")\n\t\t\treturn false, nil\n\t\t}\n\t})\n\n\tif err == wait.ErrWaitTimeout {\n\t\terr = fmt.Errorf(\"loadbalancer failed to go into ACTIVE provisioning status within allotted time\")\n\t}\n\treturn provisioningStatus, err\n}\n\n// GetDependencies returns the dependencies of the Instance task\nfunc (e *LB) GetDependencies(tasks map[string]fi.CloudupTask) []fi.CloudupTask {\n\tvar deps []fi.CloudupTask\n\tfor _, task := range tasks {\n\t\tif _, ok := task.(*Subnet); ok {\n\t\t\tdeps = append(deps, task)","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/cloudup/openstacktasks/lb.go#L61-L97","documentation":"waitLoadbalancerActiveProvisioningStatus polls Octavia GET /lbaas/loadbalancers/{id} with exponential backoff (~5 min, 22 steps). If the load balancer's provisioning_status becomes ERROR, the wait returns this error immediately instead of continuing to poll. It signals Octavia gave up provisioning the LB (amphora failed to spawn, bad subnet, quota, unsupported flavor) rather than it merely being slow.","triggerScenarios":"During `kops update cluster` LB creation, the Octavia loadbalancer transitions to provisioning_status=ERROR: amphora instance failed to boot, VIP subnet/port invalid or full, provider flavor unsupported, certificate or networking issue on the amphora, or the backend provider (e.g. ovn) rejected the config.","commonSituations":"Octavia control plane unhealthy (amphora image/version mismatch after Octavia upgrade); exhausted floating-IP or subnet IP space for the VIP; using OVN provider with unsupported LB features; control-plane node flavor too small for the amphora; cloud outage during cluster create.","solutions":["Inspect the LB and its listeners/pools: `openstack loadbalancer status show <id>` and check Octavia/amphora logs (`openstack loadbalancer amphora list`, amphora console)","Delete the ERROR-state loadbalancer and rerun `kops update cluster` to recreate it: `openstack loadbalancer delete <id>`","Verify the VIP subnet has free IPs and the amphora management network is reachable","Check Octavia service health/version compatibility (amphora image vs controller) with the cloud admin","If using a non-standard provider/flavor (e.g. ovn), confirm the cluster spec's LB provider and flavorID are supported"],"exampleFix":"# before: stuck ERROR LB from failed amphora\n# openstack loadbalancer show <lb-id>   # provisioning_status: ERROR\nopenstack loadbalancer delete <lb-id>\nkops update cluster --name <cluster> --yes\n# after: new LB reaches ACTIVE provisioning_status","handlingStrategy":"retry","validationCode":"// pre-flight: check Octavia service availability and no pre-existing ERROR LB\nfunc validateOctaviaReady(lbClient *gophercloud.ServiceClient, projectID string) error {\n\tallPages, err := loadbalancers.List(lbClient, loadbalancers.ListOpts{ProjectID: projectID}).AllPages()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"octavia API unreachable: %w\", err)\n\t}\n\tlbs, _ := loadbalancers.ExtractLoadBalancers(allPages)\n\tfor _, lb := range lbs {\n\t\tif lb.ProvisioningStatus == \"ERROR\" {\n\t\t\treturn fmt.Errorf(\"loadbalancer %s already in ERROR state; delete before applying\", lb.ID)\n\t\t}\n\t}\n\treturn nil\n}","typeGuard":"func isLBErrorState(err error) bool {\n\treturn strings.Contains(err.Error(), \"gone into ERROR state\")\n}","tryCatchPattern":"status, err := waitLoadbalancerActiveProvisioningStatus(client, lbID)\nif isLBErrorState(err) {\n\t// inspect/delete the ERROR LB, then recreate; plain retry cannot recover\n\treturn fmt.Errorf(\"LB %s entered ERROR state; check amphora logs and `openstack loadbalancer delete %s` before re-applying: %w\", lbID, lbID, err)\n}\nif errors.Is(err, wait.ErrWaitTimeout) {\n\treturn fmt.Errorf(\"LB still not ACTIVE after ~5min; check octavia health\")\n}","preventionTips":["Monitor Octavia health (amphora agents, amphora image version) before cluster applies","Ensure the VIP subnet has free IPs and the management network reaches amphorae","Delete ERROR-state load balancers promptly; retries against them cannot succeed","Verify provider/flavor (e.g. ovn) supports all LB features in the cluster spec","Watch `openstack loadbalancer status show` during applies to catch ERROR transitions early"],"tags":["openstack","octavia","loadbalancer","kops"],"backgroundTag":"loadbalancer-error-state","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}