VictoriaMetrics/VictoriaMetrics · error

unexpected status code for %q; got %d; want %d; response bod

Error message

unexpected status code for %q; got %d; want %d; response body: %q

What it means

readResponseBody expects HTTP 200 from Nova API endpoints. Any other status code produces this error including the actual code and the response body, which usually contains Nova's JSON error explanation (e.g. 401 unauthorized, 403 forbidden, 404 not found).

Source

Thrown at lib/promscrape/discovery/openstack/api.go:185

		return nil, fmt.Errorf("cannot get computeEndpoint, account doesn't have enough permissions, "+
			"availability: %s, region: %s; error: %w", cfg.availability, cfg.region, err)
	}
	return &apiCredentials{
		token:      at,
		expiration: ar.Token.ExpiresAt,
		computeURL: computeURL,
	}, nil
}

// readResponseBody reads body from http.Response.
func readResponseBody(resp *http.Response, apiURL string) ([]byte, error) {
	data, err := io.ReadAll(resp.Body)
	_ = resp.Body.Close()
	if err != nil {
		return nil, fmt.Errorf("cannot read response from %q: %w", apiURL, err)
	}
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("unexpected status code for %q; got %d; want %d; response body: %q",
			apiURL, resp.StatusCode, http.StatusOK, data)
	}
	return data, nil
}

// getAPIResponse calls openstack apiURL and returns response body.
func getAPIResponse(apiURL string, cfg *apiConfig) ([]byte, error) {
	creds, err := cfg.getFreshAPICredentials()
	if err != nil {
		return nil, err
	}
	req, err := http.NewRequest(http.MethodGet, apiURL, nil)
	if err != nil {
		return nil, fmt.Errorf("cannot create new request for openstack api url %s: %w", apiURL, err)
	}
	req.Header.Set("X-Auth-Token", creds.token)
	resp, err := cfg.client.Do(req)
	if err != nil {

View on GitHub (pinned to 5079fb58f1)

Solutions

  1. Read the response body in the error — Nova usually says exactly why (auth required, forbidden, not found)
  2. If 401: check that the token isn't expiring quickly; verify credentials and clocks (token expiry is UTC)
  3. If 403: grant the discovery user the required role (e.g. reader or member) on the project; hypervisor endpoints need elevated roles
  4. If 404: verify the compute endpoint URL/region and Nova API path still exists (API version changes)
  5. Check Nova API logs and recent policy.json/upgrades for breaking changes

Example fix

// before (role insufficient for hypervisor listing)
role: member
// after (grant elevated read role for hypervisors)
role: admin  # or 'reader' on Nova with newer default policies
Defensive patterns

Strategy: fallback

Validate before calling

# reproduce the exact API call before enabling discovery
COMPUTE_URL=$(openstack endpoint list | awk '/compute.*public/{print $10}')
curl -sS -H "X-Auth-Token: $TOKEN" "$COMPUTE_URL/servers/detail?all_tenants=true" -w '\n%{http_code}\n'
# must be 200; body explains 401/403/404 causes

Try / catch

if err != nil && strings.Contains(err.Error(), "unexpected status code") {
    switch {
    case strings.Contains(err.Error(), "got 401"), strings.Contains(err.Error(), "got 403"):
        log.Errorln("nova rejected token/role; fix credentials or grant reader role:", err)
    default:
        log.Errorln("nova api error; inspect body in error:", err)
    }
    return
}

Prevention

When it happens

Trigger: The GET to the hypervisors or servers detail URL returns a non-200 code — expired/invalid X-Auth-Token (401), insufficient role (403), wrong compute URL (404), or Nova 5xx.

Common situations: Token expired between auth and query; user lacks 'member'/'reader' role required by newer Nova API policy (401/403); SDConfig role too low for hypervisor visibility (hypervisor listing requires admin-ish roles); microversion/policy changes after a Nova upgrade.

Related errors

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/f4759c19b75a5f8c. Report an issue: GitHub.