VictoriaMetrics/VictoriaMetrics · error

cannot read GCE token from %s: %w

Error message

cannot read GCE token from %s: %w

What it means

After a successful HTTP response from the GCE-compatible metadata endpoint, readResponseBody must return the token JSON body. This error means the body could not be read: unexpected status code, empty/garbage body, or read/limit error inside readResponseBody. The endpoint is included to locate which metadata URL misbehaved.

Source

Thrown at lib/promscrape/discovery/yandexcloud/api.go:168

// getGCEInstanceCreds gets Yandex Cloud IAM token using GCE API
//
// See https://yandex.cloud/en/docs/compute/operations/vm-connect/auth-inside-vm#auth-inside-vm
func getGCEInstanceCreds(cfg *apiConfig) (*apiCredentials, error) {
	endpoint := "http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token"
	req, err := http.NewRequest(http.MethodGet, endpoint, nil)
	if err != nil {
		logger.Panicf("BUG: cannot create GCE token request for %s: %s", endpoint, err)
	}
	req.Header.Add("Metadata-Flavor", "Google")

	resp, err := cfg.client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("cannot obtain GCE token from %s: %w", endpoint, err)
	}
	data, err := readResponseBody(resp, endpoint)
	if err != nil {
		return nil, fmt.Errorf("cannot read GCE token from %s: %w", endpoint, err)
	}

	var ac gceAPICredentials
	if err := json.Unmarshal(data, &ac); err != nil {
		return nil, fmt.Errorf("cannot unmarshal GCE token from %s: %w; data=%s", endpoint, err, data)
	}
	if ac.TokenType != "Bearer" {
		return nil, fmt.Errorf("unsupported GCE token type received from %s: %q; supported: %q", endpoint, ac.TokenType, "Bearer")
	}

	expiration := time.Now().Add(time.Duration(ac.ExpiresIn) * time.Second)
	return &apiCredentials{
		Token:      ac.AccessToken,
		Expiration: expiration,
	}, nil
}

// See https://yandex.cloud/en/docs/compute/operations/vm-connect/auth-inside-vm#auth-inside-vm

View on GitHub (pinned to 5079fb58f1)

Solutions

1. Check the wrapped cause and status from readResponseBody for the concrete reason.
2. Bypass any proxy/service-mesh interception for metadata (link-local) traffic.
3. Retry — transient resets clear on the next credential refresh cycle.
4. If a body-size limit was hit, find what is returning the oversized body (usually a proxy error page).
5. Fall back to explicit yandex_passport_oauth_token if metadata remains unreliable.

client := &http.Client{Timeout: 5 * time.Second}
resp, err := client.Get(metadataEndpoint)
if err != nil {
    return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
    return fmt.Errorf("metadata endpoint returned %d", resp.StatusCode)
}

labels, err := sdc.GetLabels(baseDir)
if err != nil {
    if strings.Contains(err.Error(), "cannot read GCE token") {
        // transient metadata read failure — retry after backoff
        return retryWithBackoff(func() ([]*promutil.Labels, error) { return sdc.GetLabels(baseDir) })
    }
    return err
}

When it happens

Trigger: Metadata endpoint returned non-2xx with an unreadable/error body; response larger than the body size limit in readResponseBody; connection reset mid-body; endpoint returning empty body with 200 due to a misbehaving proxy.

Common situations: Intercepting proxies or service meshes stripping metadata responses; transient resets on the link-local interface; Yandex Cloud metadata API degraded; body size limit hit because an intermediary returned a huge HTML error page.

Related errors

• cannot obtain GCE token from %s: %w
• cannot refresh service account api token: %w
• cannot get IAM token: %w
• unexpected number of items in authToken %q; got %d; want 1 o
• cannot parse accountID from %q: %w

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/ab268b3d982e8138. Report an issue: GitHub.