VictoriaMetrics/VictoriaMetrics · error

cannot obtain availability zones: %w

Error message

cannot obtain availability zones: %w

What it means

getAvailabilityZones calls the AWS EC2 DescribeAvailabilityZones API via cfg.awsConfig.GetEC2APIResponse and wraps any failure with "cannot obtain availability zones". It is used by EC2 service discovery to build the __meta_ec2_availability_zone label map. The wrapped inner error carries the actual cause (auth, networking, throttling, etc.).

Source

Thrown at lib/promscrape/discovery/ec2/az.go:36

	azs, err := getAvailabilityZones(cfg)
	cfg.azMap = make(map[string]string, len(azs))
	if err != nil {
		logger.Warnf("couldn't load availability zones map, so __meta_ec2_availability_zone_id label isn't set: %s", err)
		return cfg.azMap
	}
	for _, az := range azs {
		cfg.azMap[az.ZoneName] = az.ZoneID
	}
	return cfg.azMap
}

func getAvailabilityZones(cfg *apiConfig) ([]AvailabilityZone, error) {
	// See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html
	azFilters := awsapi.GetFiltersQueryString(cfg.azFilters)
	data, err := cfg.awsConfig.GetEC2APIResponse("DescribeAvailabilityZones", azFilters, "")
	if err != nil {
		return nil, fmt.Errorf("cannot obtain availability zones: %w", err)
	}
	azr, err := parseAvailabilityZonesResponse(data)
	if err != nil {
		return nil, fmt.Errorf("cannot parse availability zones list: %w", err)
	}
	return azr.AvailabilityZoneInfo.Items, nil
}

// AvailabilityZonesResponse represents the response for https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html
type AvailabilityZonesResponse struct {
	AvailabilityZoneInfo AvailabilityZoneInfo `xml:"availabilityZoneInfo"`
}

// AvailabilityZoneInfo represents availabilityZoneInfo for https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html
type AvailabilityZoneInfo struct {
	Items []AvailabilityZone `xml:"item"`
}

View on GitHub (pinned to 5079fb58f1)

Solutions

1. Check AWS credentials: AWS_ACCESS_KEY_ID/SECRET or instance/IRSA profile are present and valid (aws sts get-caller-identity).
2. Grant the principal ec2:DescribeAvailabilityZones permission in the attached IAM policy.
3. Verify the region in the ec2_sd_config matches a valid AWS region and that the endpoint is reachable (network/egress, VPC endpoints).
4. Validate azFilters syntax — invalid filter names/values cause the API call to be rejected.
5. Read the wrapped inner error for the AWS error code (AuthFailure, Throttling, etc.) and address specifically.
6. Retry on throttling/backoff; transient AWS errors clear on the next discovery refresh.

Example fix

// before: IAM policy missing EC2 read
// AccessDenied: not authorized to perform ec2:DescribeAvailabilityZones
// after: add to the IAM policy
{
  "Effect": "Allow",
  "Action": ["ec2:DescribeAvailabilityZones", "ec2:DescribeInstances"],
  "Resource": "*"
}

// Pre-flight credential/permission check before enabling EC2 SD:
//   aws ec2 describe-availability-zones --region <region>
// Or programmatically:
sess := session.Must(session.NewSession())
svc := ec2.New(sess, aws.NewConfig().WithRegion("us-east-1"))
if _, err := svc.DescribeAvailabilityZones(&ec2.DescribeAvailabilityZonesInput{}); err != nil {
    log.Fatalf("EC2 AZ pre-flight failed (check creds/IAM/region): %v", err)
}

azs, err := getAvailabilityZones(cfg)
if err != nil {
    if errors.Is(err, context.DeadlineExceeded) || strings.Contains(err.Error(), "Throttling") {
        // transient: retry with exponential backoff
    } else if strings.Contains(err.Error(), "AuthFailure") || strings.Contains(err.Error(), "AccessDenied") {
        // fatal: fix credentials/IAM policy, do not hot-retry
    }
    return fmt.Errorf("ec2 AZ discovery failed: %w", err)
}

When it happens

Trigger: cfg.awsConfig.GetEC2APIResponse("DescribeAvailabilityZones", azFilters, "") fails: invalid/expired AWS credentials, missing IAM permissions (ec2:DescribeAvailabilityZones), unreachable EC2 endpoint, wrong region, or an invalid azFilters query string producing a 4xx from AWS.

Common situations: Running in an environment without AWS credentials (no env vars, no instance profile); IAM role lacking ec2:DescribeAvailabilityZones; misconfigured region in ec2_sd_config; VPC without internet/NAT egress or broken STS assume-role setup; AWS throttling (RequestLimitExceeded).

Related errors

• error when fetching instances data from EC2: %w
• unexpected status code for %q; got %d; want %d; response bod
• cannot parse availability zones list: %w
• cannot get API config: %w
• roleARN is missing for AWS_WEB_IDENTITY_TOKEN_FILE=%q; set i

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/2564abf3036f2a37. Report an issue: GitHub.