{"record":{"id":"292d4519d9ee0c91","repo":"thanos-io/thanos","slug":"failed-to-get-prometheus-version","errorCode":null,"errorMessage":"failed to get prometheus version","messagePattern":"failed to get prometheus version","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/sidecar.go","lineNumber":234,"sourceCode":"\t\t\terr := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {\n\t\t\t\titerCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)\n\t\t\t\tdefer iterCancel()\n\n\t\t\t\tif err := m.BuildVersion(iterCtx); err != nil {\n\t\t\t\t\tlevel.Warn(logger).Log(\n\t\t\t\t\t\t\"msg\", \"failed to fetch prometheus version. Is Prometheus running? Retrying\",\n\t\t\t\t\t\t\"err\", err,\n\t\t\t\t\t)\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\n\t\t\t\t\t\"msg\", \"successfully loaded prometheus version\",\n\t\t\t\t)\n\t\t\t\treturn nil\n\t\t\t})\n\t\t\tif err != nil {\n\t\t\t\treturn errors.Wrap(err, \"failed to get prometheus version\")\n\t\t\t}\n\n\t\t\t// Blocking query of external labels before joining as a Source Peer into gossip.\n\t\t\t// We retry infinitely until we reach and fetch labels from our Prometheus.\n\t\t\terr = runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {\n\t\t\t\titerCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)\n\t\t\t\tdefer iterCancel()\n\n\t\t\t\tif err := m.UpdateTimestamps(iterCtx); err != nil {\n\t\t\t\t\tlevel.Warn(logger).Log(\n\t\t\t\t\t\t\"msg\", \"failed to fetch timestamps. Is Prometheus running? Retrying\",\n\t\t\t\t\t\t\"err\", err,\n\t\t\t\t\t)\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tif err := m.UpdateLabels(iterCtx); err != nil {\n\t\t\t\t\tlevel.Warn(logger).Log(","sourceCodeStart":216,"sourceCodeEnd":252,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/sidecar.go#L216-L252","documentation":"The sidecar could not fetch the Prometheus build version via the /api/v1/status/buildinfo endpoint. runutil.Retry keeps retrying until success or ctx cancellation; if the context ends or the final attempt fails, the error is wrapped as 'failed to get prometheus version'. The version is needed to adapt API behavior (e.g. flags endpoint availability).","triggerScenarios":"errors.Wrap after the retry loop in runSidecar: promClient.BuildVersionNative(iterCtx) failed on every retry or the context was cancelled — wrong --prometheus.url, Prometheus not exposing buildinfo (proxies, very old Prometheus versions without the endpoint), timeouts.","commonSituations":"Prometheus behind an auth proxy that blocks /api/v1/status/buildinfo; Prometheus 1.x or stripped-down builds lacking the endpoint; sidecar killed before Prometheus became ready; DNS misconfiguration.","solutions":["Check the endpoint manually: curl http://<prometheus-url>/api/v1/status/buildinfo","Fix --prometheus.url and confirm DNS/network reachability from the sidecar","Upgrade or reconfigure the proxy to expose the buildinfo endpoint (requires Prometheus 2.x)","Increase --prometheus.get-config-timeout if the endpoint is slow, or investigate why Prometheus is down"],"exampleFix":"// before\niterCtx timeout 5s against slow/auth-blocked endpoint -> loop fails\n// after\n--prometheus.get-config-timeout=30s\n# plus allow /api/v1/status/buildinfo in the auth proxy config","handlingStrategy":"retry","validationCode":"resp, err := http.Get(promURL + \"/api/v1/status/buildinfo\")\nif err != nil {\n    return err\n}\nif resp.StatusCode != 200 {\n    return fmt.Errorf(\"buildinfo returned %d\", resp.StatusCode)\n}","typeGuard":null,"tryCatchPattern":"err := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {\n    return promClient.BuildVersion(iterCtx)\n})\nif err != nil {\n    return errors.Wrap(err, \"failed to get prometheus version\")\n}","preventionTips":["Ensure Prometheus 2.x so /api/v1/status/buildinfo exists","Allow the buildinfo path through auth proxies","Size --prometheus.get-config-timeout generously (e.g. 30s)","Confirm DNS resolution from the sidecar pod (nslookup)"],"tags":["prometheus","http","api","retry"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}