kubernetes/kops · error

failed to extract storage availability zones

Error message

failed to extract storage availability zones: %v

What it means

This error wraps a failure from gophercloud's az.ExtractAvailabilityZones, which parses the JSON pages returned by the Cinder (block storage) availability-zone list API into []az.AvailabilityZone structs. It is thrown inside listAvailabilityZones (upup/pkg/fi/cloudup/openstack/availability_zone.go:41) when the HTTP listing succeeded but the response body could not be decoded into the expected schema. kOps retries it with backoff via vfs.RetryWithBackoff before giving up.

Solutions

1. Inspect the raw Cinder response: curl with the same token to /v3/<project>/os-availability-zone?get_body=false style detail endpoint and check the JSON is well-formed
2. Check for proxies/load balancers in front of the Keystone/Cinder endpoints that could rewrite or truncate response bodies
3. Verify Cinder API version compatibility with the gophercloud version used by kops; upgrade gophercloud/kops if the deployment is newer
4. Retry the operation; the code already uses vfs.RetryWithBackoff, so a transient body issue may self-heal

resp, err := httpClient.Get(cinderAZURL)
if err != nil {
    return err
}
body, _ := io.ReadAll(resp.Body)
if !json.Valid(body) {
    return fmt.Errorf("cinder AZ response is not valid JSON (%d bytes)", len(body))
}
return nil

azList, err := c.ListAvailabilityZones(c.BlockStorageClient())
if err != nil {
    if strings.Contains(err.Error(), "failed to extract") {
        klog.Warningf("transient parse issue, retrying: %v", err)
        time.Sleep(2 * time.Second)
        return retry()
    }
    return err
}

When it happens

Trigger: Calling ListAvailabilityZones on the block-storage client when the Cinder availability-zone response body is malformed, truncated, or has an unexpected schema (e.g. a proxy/middlebox rewriting the response, a non-standard Cinder drop, or an HTML error page returned with 200). Also triggered persistently when every retry attempt fails extraction.

Common situations: OpenStack deployments behind API gateways that inject content into responses; very old or patched Cinder versions returning a different JSON layout; TLS-terminating proxies returning error pages with 200 status; transient network corruption that persists across the retry window.

Related errors

• error building cinder client
• Could not extract flavors
• error building dns client
• error building glance client
• error building lb client

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/c1972bf90b6813d9. Report an issue: GitHub.

Appendix: source

Thrown at upup/pkg/fi/cloudup/openstack/availability_zone.go:41

	"github.com/gophercloud/gophercloud/v2"
	az "github.com/gophercloud/gophercloud/v2/openstack/compute/v2/availabilityzones"
	"k8s.io/apimachinery/pkg/util/wait"
	"k8s.io/kops/util/pkg/vfs"
)

func (c *openstackCloud) ListAvailabilityZones(serviceClient *gophercloud.ServiceClient) (azList []az.AvailabilityZone, err error) {
	return listAvailabilityZones(c, serviceClient)
}

func listAvailabilityZones(c OpenstackCloud, serviceClient *gophercloud.ServiceClient) (azList []az.AvailabilityZone, err error) {
	done, err := vfs.RetryWithBackoff(readBackoff, func() (bool, error) {
		azPage, err := az.List(serviceClient).AllPages(context.TODO())
		if err != nil {
			return false, fmt.Errorf("failed to list storage availability zones: %v", err)
		}
		azList, err = az.ExtractAvailabilityZones(azPage)
		if err != nil {
			return false, fmt.Errorf("failed to extract storage availability zones: %v", err)
		}
		return true, nil
	})
	if !done {

		if err == nil {
			err = wait.ErrWaitTimeout
		}
		return azList, err
	}
	return azList, nil
}

func (c *openstackCloud) GetStorageAZFromCompute(computeAZ string) (*az.AvailabilityZone, error) {
	return getStorageAZFromCompute(c, computeAZ)
}

func getStorageAZFromCompute(c OpenstackCloud, computeAZ string) (*az.AvailabilityZone, error) {

View on GitHub (pinned to 4c8573c808)