{"record":{"id":"50de65ba899ec0c4","repo":"thanos-io/thanos","slug":"failed-to-get-tsdb-status-from-prometheus","errorCode":null,"errorMessage":"failed to get tsdb status from prometheus","messagePattern":"failed to get tsdb status from prometheus","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/sidecar.go","lineNumber":382,"sourceCode":"\t\t\t\t\tctx, cancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)\n\t\t\t\t\tdefer cancel()\n\n\t\t\t\t\t// Check if external labels match the provided matchers.\n\t\t\t\t\textLabels := m.Labels()\n\t\t\t\t\tpromMatchers, err := storepb.MatchersToPromMatchers(matchers...)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn nil, errors.Wrap(err, \"failed to convert matchers\")\n\t\t\t\t\t}\n\t\t\t\t\tfor _, matcher := range promMatchers {\n\t\t\t\t\t\tif !matcher.Matches(extLabels.Get(matcher.Name)) {\n\t\t\t\t\t\t\t// External labels don't match, return empty result.\n\t\t\t\t\t\t\treturn nil, nil\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\n\t\t\t\t\tstatsEntry, err := c.TSDBStatusInGRPC(ctx, conf.prometheus.url, limit)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn nil, errors.Wrap(err, \"failed to get tsdb status from prometheus\")\n\t\t\t\t\t}\n\n\t\t\t\t\treturn map[string]tsdb.Stats{\n\t\t\t\t\t\t\"\": statsEntry.ToTSDBStats(limit),\n\t\t\t\t\t}, nil\n\t\t\t\t}),\n\t\t\t),\n\t\t)\n\n\t\tstoreServer := store.NewLimitedStoreServer(store.NewInstrumentedStoreServer(reg, promStore), reg, conf.storeRateLimits)\n\t\ts := grpcserver.New(logger, reg, tracer, grpcLogOpts, logFilterMethods, comp, grpcProbe,\n\t\t\tgrpcserver.WithServer(store.RegisterStoreServer(storeServer, logger)),\n\t\t\tgrpcserver.WithServer(rules.RegisterRulesServer(rules.NewPrometheus(conf.prometheus.url, c, m.Labels))),\n\t\t\tgrpcserver.WithServer(targets.RegisterTargetsServer(targets.NewPrometheus(conf.prometheus.url, c, m.Labels))),\n\t\t\tgrpcserver.WithServer(meta.RegisterMetadataServer(meta.NewPrometheus(conf.prometheus.url, c))),\n\t\t\tgrpcserver.WithServer(exemplars.RegisterExemplarsServer(exemplarSrv)),\n\t\t\tgrpcserver.WithServer(info.RegisterInfoServer(infoSrv)),\n\t\t\tgrpcserver.WithServer(status.RegisterStatusServer(statusSrv)),","sourceCodeStart":364,"sourceCodeEnd":400,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/sidecar.go#L364-L400","documentation":"The Thanos sidecar failed to fetch TSDB status (cardinality stats) from Prometheus via the gRPC/gateway endpoint, and wraps the underlying error with this message. It is thrown inside the TSDBStatus HTTP handler in the sidecar when TSDBStatusInGRPC returns an error, meaning Prometheus was unreachable, returned a non-2xx response, or the status API call failed.","triggerScenarios":"A client requests the sidecar's /api/v1/status/tsdb endpoint; the sidecar calls TSDBStatusInGRPC against conf.prometheus.url, and that call returns any error (connection refused, timeout, non-200 from Prometheus, malformed response).","commonSituations":"Prometheus is down or restarting; the --prometheus.url points to the wrong host/port; a proxy blocks the endpoint; Prometheus is an older version without the v2 status API; network timeouts under load while collecting cardinality stats with a large limit.","solutions":["Check Prometheus is up and --prometheus.url is correct and reachable (curl <prometheus.url>/api/v1/status/tsdb).","Inspect the wrapped inner error (connection refused vs HTTP status) to determine if it is network or API-version related.","Retry the request; transient failures during Prometheus restarts resolve themselves.","Lower the limit parameter to reduce load if the endpoint times out on large TSDBs."],"exampleFix":"// before\nstats, err := sidecar.GetTSDBStatus(ctx, url, 100000)\n// after\nif err != nil {\n    level.Warn(logger).Log(\"msg\", \"tsdb status unavailable, prometheus unreachable?\", \"err\", err)\n    // fall back to cached status or retry with backoff\n}","handlingStrategy":"retry","validationCode":"resp, err := http.Get(promURL + \"/api/v1/status/tsdb?limit=1\")\nif err != nil || resp.StatusCode != 200 { /* Prometheus not ready */ }","typeGuard":null,"tryCatchPattern":"if err != nil {\n    if errors.Is(err, context.DeadlineExceeded) { /* retry with backoff */ }\n    level.Warn(logger).Log(\"msg\", \"tsdb status fetch failed\", \"err\", err)\n}","preventionTips":["Health-check Prometheus before querying the status endpoint.","Keep --prometheus.url accurate in deployment config.","Pin Prometheus to a version supporting /api/v1/status/tsdb."],"tags":["prometheus","thanos-sidecar","network","grpc"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}