thanos-io/thanos · error

request metric against

Error message

request metric against %s

What it means

LowestTimestamp performs an HTTP GET against the Prometheus /metrics endpoint. If c.Do fails at the transport level (connection refused, DNS failure, TLS error, context canceled/timeout), the error is wrapped with "request metric against <url>".

Solutions

1. Verify the Prometheus URL and that the target is reachable (curl <url>/metrics) from the Thanos host/pod.
2. Check that Prometheus is running and listening on the configured port.
3. Inspect the wrapped cause: context.DeadlineExceeded means increase timeout or fix slow scrape; connection refused means wrong address.
4. Check network policies, service DNS names, and TLS settings between Thanos and Prometheus.

Example fix

// before
resp, err := c.Do(req.WithContext(ctx))
if err != nil { return 0, errors.Wrapf(err, "request metric against %s", u.String()) }

// after
// Pre-check reachability and use a bounded context
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if err := runutil.Retry(3, 2*time.Second, func() error {
	_, err := http.Get(u.String())
	return err
}); err != nil {
	return err // surface persistent unreachability early
}

// pre-check reachability
if err := checkTCPReachable(host, port, 3*time.Second); err != nil {
	return fmt.Errorf("prometheus %s unreachable: %w", addr, err)
}

if err := runutil.Retry(3, 2*time.Second, func() error {
	_, err := client.LowestTimestamp(ctx, base)
	return err
}); err != nil {
	level.Warn(logger).Log("msg", "lowest timestamp unavailable", "err", err)
}

When it happens

Trigger: c.Do(req.WithContext(ctx)) errors: target Prometheus is down/unreachable, wrong host:port, TLS handshake failure, or the passed ctx is canceled/deadline-exceeded.

Common situations: Sidecar cannot reach the local Prometheus (wrong --prometheus.url), Prometheus restarted or crashed, network policy/firewall blocking the port, Kubernetes DNS issues, or query fan-out timing out.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• read query range response
• send request
• failed to validate prometheus flags
• perform request against
• request config against

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/0a899c0ab431c334. Report an issue: GitHub.

Appendix: source

Thrown at pkg/promclient/promclient.go:715

// LowestTimestamp returns the lowest timestamp in the TSDB by parsing the /metrics endpoint
// and extracting the prometheus_tsdb_lowest_timestamp_seconds metric from it.
func (c *Client) LowestTimestamp(ctx context.Context, base *url.URL) (int64, error) {
	u := *base
	u.Path = path.Join(u.Path, "/metrics")

	level.Debug(c.logger).Log("msg", "lowest timestamp", "url", u.String())

	req, err := http.NewRequest(http.MethodGet, u.String(), nil)
	if err != nil {
		return 0, errors.Wrap(err, "create request")
	}

	span, ctx := tracing.StartSpan(ctx, "/lowest_timestamp HTTP[client]")
	defer span.Finish()

	resp, err := c.Do(req.WithContext(ctx))
	if err != nil {
		return 0, errors.Wrapf(err, "request metric against %s", u.String())
	}
	defer runutil.ExhaustCloseWithLogOnErr(c.logger, resp.Body, "request body")

	parser := expfmt.NewTextParser(model.UTF8Validation)
	families, err := parser.TextToMetricFamilies(resp.Body)
	if err != nil {
		return 0, errors.Wrapf(err, "parsing metric families against %s", u.String())
	}
	mf, ok := families["prometheus_tsdb_lowest_timestamp_seconds"]
	if !ok {
		return 0, errors.Wrapf(err, "metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'")
	}
	val := 1000 * mf.GetMetric()[0].GetGauge().GetValue()

	// in the case that we dont have cut a block yet, TSDB lowest timestamp is math.MaxInt64
	// but its represented as float and truncated so we need to do this weird comparison.
	// Since we use this for fan-out pruning we use min timestamp here to include this prometheus.
	if val == float64(math.MaxInt64) {

View on GitHub (pinned to 35b8b99117)