{"record":{"id":"c1972bf90b6813d9","repo":"kubernetes/kops","slug":"failed-to-extract-storage-availability-zones-v","errorCode":null,"errorMessage":"failed to extract storage availability zones: %v","messagePattern":"failed to extract storage availability zones: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/cloudup/openstack/availability_zone.go","lineNumber":41,"sourceCode":"\t\"github.com/gophercloud/gophercloud/v2\"\n\taz \"github.com/gophercloud/gophercloud/v2/openstack/compute/v2/availabilityzones\"\n\t\"k8s.io/apimachinery/pkg/util/wait\"\n\t\"k8s.io/kops/util/pkg/vfs\"\n)\n\nfunc (c *openstackCloud) ListAvailabilityZones(serviceClient *gophercloud.ServiceClient) (azList []az.AvailabilityZone, err error) {\n\treturn listAvailabilityZones(c, serviceClient)\n}\n\nfunc listAvailabilityZones(c OpenstackCloud, serviceClient *gophercloud.ServiceClient) (azList []az.AvailabilityZone, err error) {\n\tdone, err := vfs.RetryWithBackoff(readBackoff, func() (bool, error) {\n\t\tazPage, err := az.List(serviceClient).AllPages(context.TODO())\n\t\tif err != nil {\n\t\t\treturn false, fmt.Errorf(\"failed to list storage availability zones: %v\", err)\n\t\t}\n\t\tazList, err = az.ExtractAvailabilityZones(azPage)\n\t\tif err != nil {\n\t\t\treturn false, fmt.Errorf(\"failed to extract storage availability zones: %v\", err)\n\t\t}\n\t\treturn true, nil\n\t})\n\tif !done {\n\n\t\tif err == nil {\n\t\t\terr = wait.ErrWaitTimeout\n\t\t}\n\t\treturn azList, err\n\t}\n\treturn azList, nil\n}\n\nfunc (c *openstackCloud) GetStorageAZFromCompute(computeAZ string) (*az.AvailabilityZone, error) {\n\treturn getStorageAZFromCompute(c, computeAZ)\n}\n\nfunc getStorageAZFromCompute(c OpenstackCloud, computeAZ string) (*az.AvailabilityZone, error) {","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/cloudup/openstack/availability_zone.go#L23-L59","documentation":"This error wraps a failure from gophercloud's az.ExtractAvailabilityZones, which parses the JSON pages returned by the Cinder (block storage) availability-zone list API into []az.AvailabilityZone structs. It is thrown inside listAvailabilityZones (upup/pkg/fi/cloudup/openstack/availability_zone.go:41) when the HTTP listing succeeded but the response body could not be decoded into the expected schema. kOps retries it with backoff via vfs.RetryWithBackoff before giving up.","triggerScenarios":"Calling ListAvailabilityZones on the block-storage client when the Cinder availability-zone response body is malformed, truncated, or has an unexpected schema (e.g. a proxy/middlebox rewriting the response, a non-standard Cinder drop, or an HTML error page returned with 200). Also triggered persistently when every retry attempt fails extraction.","commonSituations":"OpenStack deployments behind API gateways that inject content into responses; very old or patched Cinder versions returning a different JSON layout; TLS-terminating proxies returning error pages with 200 status; transient network corruption that persists across the retry window.","solutions":["Inspect the raw Cinder response: curl with the same token to /v3/<project>/os-availability-zone?get_body=false style detail endpoint and check the JSON is well-formed","Check for proxies/load balancers in front of the Keystone/Cinder endpoints that could rewrite or truncate response bodies","Verify Cinder API version compatibility with the gophercloud version used by kops; upgrade gophercloud/kops if the deployment is newer","Retry the operation; the code already uses vfs.RetryWithBackoff, so a transient body issue may self-heal"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"resp, err := httpClient.Get(cinderAZURL)\nif err != nil {\n    return err\n}\nbody, _ := io.ReadAll(resp.Body)\nif !json.Valid(body) {\n    return fmt.Errorf(\"cinder AZ response is not valid JSON (%d bytes)\", len(body))\n}\nreturn nil","typeGuard":null,"tryCatchPattern":"azList, err := c.ListAvailabilityZones(c.BlockStorageClient())\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to extract\") {\n        klog.Warningf(\"transient parse issue, retrying: %v\", err)\n        time.Sleep(2 * time.Second)\n        return retry()\n    }\n    return err\n}","preventionTips":["Keep gophercloud and kops versions compatible with your Cinder release","Avoid proxies that rewrite API response bodies","Monitor for Cinder API changes when upgrading OpenStack","Validate Keystone/Cinder endpoints return well-formed JSON with curl before cluster operations"],"tags":["openstack","cinder","json-parsing","gophercloud"],"backgroundTag":"api-response-parse-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}