metric families did not contain…

Error message

metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'

What it means

The /metrics response parsed successfully but did not expose the prometheus_tsdb_lowest_timestamp_seconds metric family, which LowestTimestamp requires to compute the oldest sample time. Note the code wraps `err`, which is nil here, so the message is the whole error.

Solutions

Verify the URL points at the main Prometheus process and curl /metrics | grep prometheus_tsdb_lowest_timestamp_seconds to confirm the metric exists.
Check Prometheus is not running in agent mode or with TSDB metrics filtered out; upgrade or reconfigure if so.
Fix the configured base URL if it points at a proxy/other exporter.
If you cannot guarantee the metric, guard callers of LowestTimestamp to tolerate the error instead of failing store startup.

Example fix

// before
mf, ok := families["prometheus_tsdb_lowest_timestamp_seconds"]
if !ok {
	return 0, errors.Wrapf(err, "metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'")
}

// after
mf, ok := families["prometheus_tsdb_lowest_timestamp_seconds"]
if !ok {
	return 0, errors.Errorf("metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds' at %s", u.String())
}

Defensive patterns

Strategy: validation

Validate before calling

out, err := exec.Command("curl", "-sf", metricsURL).Output()
if err != nil || !strings.Contains(string(out), "prometheus_tsdb_lowest_timestamp_seconds") {
	return fmt.Errorf("%s does not expose prometheus_tsdb_lowest_timestamp_seconds", metricsURL)
}

Prevention

Verify the target is a full Prometheus server (not agent mode or another exporter)
grep /metrics for required metric families during deployment checks
Pin Prometheus versions known to export TSDB metrics used by Thanos

When it happens

Trigger: families["prometheus_tsdb_lowest_timestamp_seconds"] lookup misses — the queried endpoint is not a real Prometheus TSDB (e.g. a Thanos sidecar, Alertmanager, or generic exporter on that URL) or a Prometheus version/edition that does not export this metric (e.g. prometheus.tsdb Exemplar/agent mode or metrics disabled).

Common situations: Pointing Thanos sidecar at the wrong service (proxy or agent-mode Prometheus); Prometheus started with --web.enable-metrics restricted or agent mode where TSDB head metrics differ; scrape path misconfigured so a different endpoint's output is parsed.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a94c5e468160f28e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/promclient/promclient.go:726

	}

	span, ctx := tracing.StartSpan(ctx, "/lowest_timestamp HTTP[client]")
	defer span.Finish()

	resp, err := c.Do(req.WithContext(ctx))
	if err != nil {
		return 0, errors.Wrapf(err, "request metric against %s", u.String())
	}
	defer runutil.ExhaustCloseWithLogOnErr(c.logger, resp.Body, "request body")

	parser := expfmt.NewTextParser(model.UTF8Validation)
	families, err := parser.TextToMetricFamilies(resp.Body)
	if err != nil {
		return 0, errors.Wrapf(err, "parsing metric families against %s", u.String())
	}
	mf, ok := families["prometheus_tsdb_lowest_timestamp_seconds"]
	if !ok {
		return 0, errors.Wrapf(err, "metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'")
	}
	val := 1000 * mf.GetMetric()[0].GetGauge().GetValue()

	// in the case that we dont have cut a block yet, TSDB lowest timestamp is math.MaxInt64
	// but its represented as float and truncated so we need to do this weird comparison.
	// Since we use this for fan-out pruning we use min timestamp here to include this prometheus.
	if val == float64(math.MaxInt64) {
		return math.MinInt64, nil
	}
	return int64(val), nil
}

func formatTime(t time.Time) string {
	return strconv.FormatFloat(float64(t.Unix())+float64(t.Nanosecond())/1e9, 'f', -1, 64)
}

func (c *Client) get2xxResultWithGRPCErrors(ctx context.Context, spanName string, u *url.URL, data any) error {
	span, ctx := tracing.StartSpan(ctx, spanName)

View on GitHub (pinned to 35b8b99117)