{"record":{"id":"a94c5e468160f28e","repo":"thanos-io/thanos","slug":"metric-families-did-not-contain-prometheus-tsdb-l","errorCode":null,"errorMessage":"metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'","messagePattern":"metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/promclient/promclient.go","lineNumber":726,"sourceCode":"\t}\n\n\tspan, ctx := tracing.StartSpan(ctx, \"/lowest_timestamp HTTP[client]\")\n\tdefer span.Finish()\n\n\tresp, err := c.Do(req.WithContext(ctx))\n\tif err != nil {\n\t\treturn 0, errors.Wrapf(err, \"request metric against %s\", u.String())\n\t}\n\tdefer runutil.ExhaustCloseWithLogOnErr(c.logger, resp.Body, \"request body\")\n\n\tparser := expfmt.NewTextParser(model.UTF8Validation)\n\tfamilies, err := parser.TextToMetricFamilies(resp.Body)\n\tif err != nil {\n\t\treturn 0, errors.Wrapf(err, \"parsing metric families against %s\", u.String())\n\t}\n\tmf, ok := families[\"prometheus_tsdb_lowest_timestamp_seconds\"]\n\tif !ok {\n\t\treturn 0, errors.Wrapf(err, \"metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'\")\n\t}\n\tval := 1000 * mf.GetMetric()[0].GetGauge().GetValue()\n\n\t// in the case that we dont have cut a block yet, TSDB lowest timestamp is math.MaxInt64\n\t// but its represented as float and truncated so we need to do this weird comparison.\n\t// Since we use this for fan-out pruning we use min timestamp here to include this prometheus.\n\tif val == float64(math.MaxInt64) {\n\t\treturn math.MinInt64, nil\n\t}\n\treturn int64(val), nil\n}\n\nfunc formatTime(t time.Time) string {\n\treturn strconv.FormatFloat(float64(t.Unix())+float64(t.Nanosecond())/1e9, 'f', -1, 64)\n}\n\nfunc (c *Client) get2xxResultWithGRPCErrors(ctx context.Context, spanName string, u *url.URL, data any) error {\n\tspan, ctx := tracing.StartSpan(ctx, spanName)","sourceCodeStart":708,"sourceCodeEnd":744,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/promclient/promclient.go#L708-L744","documentation":"The /metrics response parsed successfully but did not expose the prometheus_tsdb_lowest_timestamp_seconds metric family, which LowestTimestamp requires to compute the oldest sample time. Note the code wraps `err`, which is nil here, so the message is the whole error.","triggerScenarios":"families[\"prometheus_tsdb_lowest_timestamp_seconds\"] lookup misses — the queried endpoint is not a real Prometheus TSDB (e.g. a Thanos sidecar, Alertmanager, or generic exporter on that URL) or a Prometheus version/edition that does not export this metric (e.g. prometheus.tsdb Exemplar/agent mode or metrics disabled).","commonSituations":"Pointing Thanos sidecar at the wrong service (proxy or agent-mode Prometheus); Prometheus started with --web.enable-metrics restricted or agent mode where TSDB head metrics differ; scrape path misconfigured so a different endpoint's output is parsed.","solutions":["Verify the URL points at the main Prometheus process and curl /metrics | grep prometheus_tsdb_lowest_timestamp_seconds to confirm the metric exists.","Check Prometheus is not running in agent mode or with TSDB metrics filtered out; upgrade or reconfigure if so.","Fix the configured base URL if it points at a proxy/other exporter.","If you cannot guarantee the metric, guard callers of LowestTimestamp to tolerate the error instead of failing store startup."],"exampleFix":"// before\nmf, ok := families[\"prometheus_tsdb_lowest_timestamp_seconds\"]\nif !ok {\n\treturn 0, errors.Wrapf(err, \"metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds'\")\n}\n\n// after\nmf, ok := families[\"prometheus_tsdb_lowest_timestamp_seconds\"]\nif !ok {\n\treturn 0, errors.Errorf(\"metric families did not contain 'prometheus_tsdb_lowest_timestamp_seconds' at %s\", u.String())\n}","handlingStrategy":"validation","validationCode":"out, err := exec.Command(\"curl\", \"-sf\", metricsURL).Output()\nif err != nil || !strings.Contains(string(out), \"prometheus_tsdb_lowest_timestamp_seconds\") {\n\treturn fmt.Errorf(\"%s does not expose prometheus_tsdb_lowest_timestamp_seconds\", metricsURL)\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Verify the target is a full Prometheus server (not agent mode or another exporter)","grep /metrics for required metric families during deployment checks","Pin Prometheus versions known to export TSDB metrics used by Thanos"],"tags":["prometheus","metrics","configuration","missing-metric"],"backgroundTag":"missing-required-argument","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}