kubernetes/kops · error

getting location for bucket %q: %w

Error message

getting location for bucket %q: %w

What it means

bucketLocationViaHead resolves an S3 bucket's region via HeadBucket. On any HeadBucket failure where the region could not be recovered from the x-amz-bucket-region response header (e.g. 403 Forbidden with no such header, 404 NoSuchBucket, network failure, or a client-side error that isn't a smithy ResponseError), it wraps the underlying AWS SDK error and throws this error. It means kOps could not determine which region the state-store bucket lives in, so the S3 VFS context cannot be built.

Source

Thrown at util/pkg/vfs/s3context.go:309

	defer span.End()

	out, err := s3Client.HeadBucket(ctx, &s3.HeadBucketInput{
		Bucket: aws.String(bucket),
	})
	if err == nil {
		if out.BucketRegion != nil && *out.BucketRegion != "" {
			return *out.BucketRegion, nil
		}
		return "", fmt.Errorf("HeadBucket on %q did not return a bucket region", bucket)
	}

	var respErr *smithyhttp.ResponseError
	if errors.As(err, &respErr) && respErr.Response != nil && respErr.Response.Response != nil {
		if bucketRegion := respErr.Response.Header.Get("x-amz-bucket-region"); bucketRegion != "" {
			return bucketRegion, nil
		}
	}
	return "", fmt.Errorf("getting location for bucket %q: %w", bucket, err)
}

// isRunningOnEC2 determines if we could be running on EC2.
// It is used to avoid a call to the metadata service to get the current region,
// because that call is slow if not running on EC2
func isRunningOnEC2(ctx context.Context) (bool, error) {
	if runtime.GOOS == "linux" {
		// Approach based on https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html
		productUUID, err := os.ReadFile("/sys/devices/virtual/dmi/id/product_uuid")
		if err != nil {
			klog.V(2).Infof("unable to read /sys/devices/virtual/dmi/id/product_uuid, assuming not running on EC2: %v", err)
			return false, nil
		}

		s := strings.ToLower(strings.TrimSpace(string(productUUID)))
		if strings.HasPrefix(s, "ec2") {
			klog.V(2).Infof("product_uuid is %q, assuming running on EC2", s)
			return true, nil

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify the bucket exists and the name is spelled correctly: aws s3api head-bucket --bucket <name> — reproduce the same 404/403 locally
2. Fix credentials/permissions: grant s3:ListBucket (HeadBucket) on the bucket to the calling principal, or fix AWS_PROFILE / credential chain
3. Check network reachability to the S3 endpoint (proxy, VPC endpoint, DNS); test with curl -v https://s3.amazonaws.com/<bucket>
4. If the bucket moved or the state store was changed, run kops update cluster / export with the correct --state store value
5. Retry on transient network errors; if region ambiguity persists, set the region explicitly in cluster config or AWS_REGION so a wrong-region client isn't used

Example fix

// before: no permission on cross-account bucket
$ kops get cluster --state s3://our-cluster-state
error: getting location for bucket "our-cluster-state": operation error S3: HeadBucket, https response error StatusCode: 403
// after: add HeadBucket permission
{
  "Effect": "Allow",
  "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
  "Resource": "arn:aws:s3:::our-cluster-state"
}

region, err := bucketLocationViaHead(ctx, client, bucket)
if err != nil {
	// fall back to GetBucketLocation API before giving up
	loc, lerr := client.GetBucketLocation(ctx, &s3.GetBucketLocationInput{Bucket: aws.String(bucket)})
	if lerr == nil && loc.LocationConstraint != "" {
		region = string(loc.LocationConstraint)
	} else {
		return fmt.Errorf("getting location for bucket %q: %w", bucket, err)
	}
}

When it happens

Trigger: s3Client.HeadBucket fails and either (a) errors.As(err, *smithyhttp.ResponseError) is false (network error, DNS failure, credential/sts error, request signing failure), (b) the response error carries no x-amz-bucket-region header (404 for a nonexistent/deleted bucket, 403 without region info, connection reset), or (c) the header is present but empty. Triggered during getDetailsForBucket when initializing the S3 VFS context for a cluster state store.

Common situations: Typo in the state store bucket name or bucket deleted before running kops; IAM credentials lack s3:ListBucket/HeadBucket permission on the bucket; network egress/firewall or VPC endpoint misconfiguration blocking s3.<region>.amazonaws.com; DNS problems in restricted clusters (air-gapped environments); using a bucket name that exists in another account you can't access.

Related errors

• checking if bucket was public: %w
• error deleting %s: %v
• error reading %s: %v
• error listing %s: %v
• failed to get bucket details for %q: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/d6d72bc5489bf853. Report an issue: GitHub.