thanos-io/thanos · error

send request

Error message

send request

What it means

startPromRemoteRead performs the POST via p.client.Do(preq.WithContext(ctx)); any transport-level failure (connection refused, DNS failure, timeout, context cancellation) is wrapped as "send request". It means the HTTP request to Prometheus's remote read endpoint never completed successfully.

Solutions

1. Confirm the Prometheus address is reachable: curl http://<addr>/api/v1/read (expect non-connection-error)
2. Check for context cancellation — increase caller/step timeouts if the query legitimately takes long
3. Verify TLS configuration (CA bundle, client certs) if using https
4. Check DNS/service discovery so the address resolves to a live instance

Example fix

// before: no per-query timeout control
cctx := ctx
// after: bound remote-read latency explicitly
ctx2, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
presp, err = p.client.Do(preq.WithContext(ctx2))

conn, err := net.DialTimeout("tcp", host, 2*time.Second)
if err != nil { return fmt.Errorf("prometheus unreachable: %w", err) }
conn.Close()

var resp *storepb.SeriesResponse
backoff := 100 * time.Millisecond
for i := 0; i < 3; i++ {
    resp, err = store.Series(ctx, req)
    if err != nil && strings.Contains(err.Error(), "send request") && ctx.Err() == nil {
        time.Sleep(backoff); backoff *= 2; continue
    }
    break
}

When it happens

Trigger: Prometheus unreachable at the configured address, TLS handshake failure, network partition, or the caller's context being canceled/timed out mid-request.

Common situations: Wrong host/port in store address; Prometheus not running or behind a firewall; mTLS/TLS cert issues; slow queries exceeding client timeout; query canceled by upstream (common in multi-tier Thanos setups).

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• read query range response
• request metric against
• failed to validate prometheus flags
• perform request against
• request config against

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9b5a84459362e805. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/prometheus.go:481

	if err != nil {
		return nil, errors.Wrap(err, "marshal read request")
	}

	u := *p.base
	u.Path = path.Join(u.Path, "api/v1/read")

	preq, err := http.NewRequest("POST", u.String(), bytes.NewReader(snappy.Encode(nil, reqb)))
	if err != nil {
		return nil, errors.Wrap(err, "unable to create request")
	}
	preq.Header.Add("Content-Encoding", "snappy")
	preq.Header.Set("Content-Type", "application/x-stream-protobuf")
	preq.Header.Set("X-Prometheus-Remote-Read-Version", "0.1.0")

	preq.Header.Set("User-Agent", clientconfig.ThanosUserAgent)
	presp, err = p.client.Do(preq.WithContext(ctx))
	if err != nil {
		return nil, errors.Wrap(err, "send request")
	}
	if presp.StatusCode/100 != 2 {
		// Best effort read.
		b, err := io.ReadAll(presp.Body)
		if err != nil {
			level.Error(p.logger).Log("msg", "failed to read response from non 2XX remote read request", "err", err)
		}
		_ = presp.Body.Close()
		return nil, errors.Errorf("request failed with code %s; msg %s", presp.Status, string(b))
	}

	return presp, nil
}

// matchesExternalLabels returns false if given matchers are not matching external labels.
// If true, matchesExternalLabels also returns Prometheus matchers without those matching external labels.
func matchesExternalLabels(ms []storepb.LabelMatcher, externalLabels labels.Labels, cache storecache.MatchersCache) (bool, []*labels.Matcher, error) {
	var (

View on GitHub (pinned to 35b8b99117)