unexpected status code returned when scraping %q: %d; expect

Error message

unexpected status code returned when scraping %q: %d; expecting %d; response body: %q

What it means

Raised in client.ReadData when the scrape response status is not 200: the body (truncated to max_scrape_size) is included to aid debugging, and a vm_promscrape_scrapes_total{status_code=...} counter is incremented.

Source

Thrown at lib/promscrape/client.go:181

	scrapeRequests.Inc()
	resp, err := c.c.Do(req)
	if err != nil {
		if ue, ok := err.(*url.Error); ok && ue.Timeout() {
			scrapesTimedout.Inc()
		}
		return false, fmt.Errorf("cannot perform request to %q: %w", c.scrapeURL, err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		metrics.GetOrCreateCounter(fmt.Sprintf(`vm_promscrape_scrapes_total{status_code="%d"}`, resp.StatusCode)).Inc()
		lr := ioutil.GetLimitedReader(resp.Body, c.maxScrapeSize+1)
		respBody, err := io.ReadAll(lr)
		ioutil.PutLimitedReader(lr)
		if err != nil {
			respBody = []byte(err.Error())
		}
		return false, fmt.Errorf("unexpected status code returned when scraping %q: %d; expecting %d; response body: %q",
			c.scrapeURL, resp.StatusCode, http.StatusOK, respBody)
	}
	scrapesOK.Inc()

	// Read the data from resp.Body
	lr := ioutil.GetLimitedReader(resp.Body, c.maxScrapeSize+1)
	_, err = dst.ReadFrom(lr)
	ioutil.PutLimitedReader(lr)
	if err != nil {
		if ue, ok := err.(*url.Error); ok && ue.Timeout() {
			scrapesTimedout.Inc()
		}
		return false, fmt.Errorf("cannot read data from %s: %w", c.scrapeURL, err)
	}
	if int64(dst.Len()) > c.maxScrapeSize {
		maxScrapeSizeExceeded.Inc()
		return false, fmt.Errorf("the response from %q exceeds -promscrape.maxScrapeSize or max_scrape_size in the scrape config (%d bytes). "+
			"Possible solutions are: reduce the response size for the target, increase -promscrape.maxScrapeSize command-line flag, "+

View on GitHub (pinned to 5079fb58f1)

Solutions

Read the response body snippet for the target's error message
Common causes: wrong metrics_path (404), auth required (401), target behind a proxy (403/502) — fix job config accordingly

Defensive patterns

Strategy: validation

When it happens

Trigger: Thrown at lib/promscrape/client.go:181 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/381ebba2413c4794. Report an issue: GitHub.