{"record":{"id":"2564abf3036f2a37","repo":"VictoriaMetrics/VictoriaMetrics","slug":"cannot-obtain-availability-zones-w","errorCode":null,"errorMessage":"cannot obtain availability zones: %w","messagePattern":"cannot obtain availability zones: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/promscrape/discovery/ec2/az.go","lineNumber":36,"sourceCode":"\n\tazs, err := getAvailabilityZones(cfg)\n\tcfg.azMap = make(map[string]string, len(azs))\n\tif err != nil {\n\t\tlogger.Warnf(\"couldn't load availability zones map, so __meta_ec2_availability_zone_id label isn't set: %s\", err)\n\t\treturn cfg.azMap\n\t}\n\tfor _, az := range azs {\n\t\tcfg.azMap[az.ZoneName] = az.ZoneID\n\t}\n\treturn cfg.azMap\n}\n\nfunc getAvailabilityZones(cfg *apiConfig) ([]AvailabilityZone, error) {\n\t// See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html\n\tazFilters := awsapi.GetFiltersQueryString(cfg.azFilters)\n\tdata, err := cfg.awsConfig.GetEC2APIResponse(\"DescribeAvailabilityZones\", azFilters, \"\")\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"cannot obtain availability zones: %w\", err)\n\t}\n\tazr, err := parseAvailabilityZonesResponse(data)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"cannot parse availability zones list: %w\", err)\n\t}\n\treturn azr.AvailabilityZoneInfo.Items, nil\n}\n\n// AvailabilityZonesResponse represents the response for https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html\ntype AvailabilityZonesResponse struct {\n\tAvailabilityZoneInfo AvailabilityZoneInfo `xml:\"availabilityZoneInfo\"`\n}\n\n// AvailabilityZoneInfo represents availabilityZoneInfo for https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeAvailabilityZones.html\ntype AvailabilityZoneInfo struct {\n\tItems []AvailabilityZone `xml:\"item\"`\n}\n","sourceCodeStart":18,"sourceCodeEnd":54,"githubUrl":"https://github.com/VictoriaMetrics/VictoriaMetrics/blob/5079fb58f1e8e62113f90c945ad71586c797d770/lib/promscrape/discovery/ec2/az.go#L18-L54","documentation":"getAvailabilityZones calls the AWS EC2 DescribeAvailabilityZones API via cfg.awsConfig.GetEC2APIResponse and wraps any failure with \"cannot obtain availability zones\". It is used by EC2 service discovery to build the __meta_ec2_availability_zone label map. The wrapped inner error carries the actual cause (auth, networking, throttling, etc.).","triggerScenarios":"cfg.awsConfig.GetEC2APIResponse(\"DescribeAvailabilityZones\", azFilters, \"\") fails: invalid/expired AWS credentials, missing IAM permissions (ec2:DescribeAvailabilityZones), unreachable EC2 endpoint, wrong region, or an invalid azFilters query string producing a 4xx from AWS.","commonSituations":"Running in an environment without AWS credentials (no env vars, no instance profile); IAM role lacking ec2:DescribeAvailabilityZones; misconfigured region in ec2_sd_config; VPC without internet/NAT egress or broken STS assume-role setup; AWS throttling (RequestLimitExceeded).","solutions":["Check AWS credentials: AWS_ACCESS_KEY_ID/SECRET or instance/IRSA profile are present and valid (aws sts get-caller-identity).","Grant the principal ec2:DescribeAvailabilityZones permission in the attached IAM policy.","Verify the region in the ec2_sd_config matches a valid AWS region and that the endpoint is reachable (network/egress, VPC endpoints).","Validate azFilters syntax — invalid filter names/values cause the API call to be rejected.","Read the wrapped inner error for the AWS error code (AuthFailure, Throttling, etc.) and address specifically.","Retry on throttling/backoff; transient AWS errors clear on the next discovery refresh."],"exampleFix":"// before: IAM policy missing EC2 read\n// AccessDenied: not authorized to perform ec2:DescribeAvailabilityZones\n// after: add to the IAM policy\n{\n  \"Effect\": \"Allow\",\n  \"Action\": [\"ec2:DescribeAvailabilityZones\", \"ec2:DescribeInstances\"],\n  \"Resource\": \"*\"\n}","handlingStrategy":"try-catch","validationCode":"// Pre-flight credential/permission check before enabling EC2 SD:\n//   aws ec2 describe-availability-zones --region <region>\n// Or programmatically:\nsess := session.Must(session.NewSession())\nsvc := ec2.New(sess, aws.NewConfig().WithRegion(\"us-east-1\"))\nif _, err := svc.DescribeAvailabilityZones(&ec2.DescribeAvailabilityZonesInput{}); err != nil {\n    log.Fatalf(\"EC2 AZ pre-flight failed (check creds/IAM/region): %v\", err)\n}","typeGuard":null,"tryCatchPattern":"azs, err := getAvailabilityZones(cfg)\nif err != nil {\n    if errors.Is(err, context.DeadlineExceeded) || strings.Contains(err.Error(), \"Throttling\") {\n        // transient: retry with exponential backoff\n    } else if strings.Contains(err.Error(), \"AuthFailure\") || strings.Contains(err.Error(), \"AccessDenied\") {\n        // fatal: fix credentials/IAM policy, do not hot-retry\n    }\n    return fmt.Errorf(\"ec2 AZ discovery failed: %w\", err)\n}","preventionTips":["Attach ec2:DescribeAvailabilityZones (and DescribeInstances) to the discovery IAM role.","Set credentials via instance profile/IRSA instead of static keys that expire.","Pin and validate the region in ec2_sd_config; ensure VPC endpoints/egress allow EC2 calls.","Keep azFilters minimal and syntactically valid (awsapi.GetFiltersQueryString format).","Run `aws sts get-caller-identity` in the scraper environment to confirm identity."],"tags":["aws","ec2","service-discovery","iam"],"backgroundTag":"aws-api-call-failed","analyzedSha":"5079fb58f1e8e62113f90c945ad71586c797d770","analyzedAt":"2026-09-03T18:10:26.153Z","contentChangedAt":"2026-09-03T18:10:26.153Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}