{"record":{"id":"6033a169a57b40dd","repo":"thanos-io/thanos","slug":"failed-to-validate-prometheus-flags","errorCode":null,"errorMessage":"failed to validate prometheus flags","messagePattern":"failed to validate prometheus flags","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/sidecar.go","lineNumber":211,"sourceCode":"\t\t\t\terr := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {\n\t\t\t\t\titerCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)\n\t\t\t\t\tdefer iterCancel()\n\n\t\t\t\t\tif err := validatePrometheus(iterCtx, m.client, logger, &conf, m); err != nil {\n\t\t\t\t\t\tlevel.Warn(logger).Log(\n\t\t\t\t\t\t\t\"msg\", \"failed to validate prometheus flags. Is Prometheus running? Retrying\",\n\t\t\t\t\t\t\t\"err\", err,\n\t\t\t\t\t\t)\n\t\t\t\t\t\treturn err\n\t\t\t\t\t}\n\n\t\t\t\t\tlevel.Info(logger).Log(\n\t\t\t\t\t\t\"msg\", \"successfully validated prometheus flags\",\n\t\t\t\t\t)\n\t\t\t\t\treturn nil\n\t\t\t\t})\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"failed to validate prometheus flags\")\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// We retry infinitely until we reach and fetch BuildVersion from our Prometheus.\n\t\t\terr := runutil.Retry(conf.prometheus.getConfigInterval, ctx.Done(), func() error {\n\t\t\t\titerCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)\n\t\t\t\tdefer iterCancel()\n\n\t\t\t\tif err := m.BuildVersion(iterCtx); err != nil {\n\t\t\t\t\tlevel.Warn(logger).Log(\n\t\t\t\t\t\t\"msg\", \"failed to fetch prometheus version. Is Prometheus running? Retrying\",\n\t\t\t\t\t\t\"err\", err,\n\t\t\t\t\t)\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\n\t\t\t\t\t\"msg\", \"successfully loaded prometheus version\",","sourceCodeStart":193,"sourceCodeEnd":229,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/sidecar.go#L193-L229","documentation":"The periodic Prometheus flags validation (checking --prometheus.url is reachable and flags such as external-labels/TSDB settings are consistent) returned an error, wrapped as 'failed to validate prometheus flags'. The sidecar retries validation in the background; if a single validation attempt fails, the underlying client error is wrapped here and reported by run.Group.","triggerScenarios":"errors.Wrap around the validation goroutine in runSidecar: the inner func returned an error from promClient.BuildVersion / flag checks — typically because --prometheus.url is wrong or unreachable, the endpoint is not a Prometheus server, or a transient HTTP failure occurred in the non-retried validation path.","commonSituations":"Service DNS name wrong in Kubernetes; Prometheus on a different port than 9090; TLS/proxy in front of Prometheus altering paths; Prometheus down during sidecar startup.","solutions":["Confirm --prometheus.url (scheme/host/port) resolves to the Prometheus HTTP API from the sidecar pod (curl <url>/api/v1/status/buildinfo)","Retry the sidecar if Prometheus was temporarily down during startup","Remove any proxy that mangles /api/v1 paths or point directly at the Prometheus service","Check sidecar→Prometheus network policies/firewall rules"],"exampleFix":"// before\n--prometheus.url=http://prometheus:9091   # wrong port\n// after\n--prometheus.url=http://prometheus.monitoring.svc.cluster.local:9090","handlingStrategy":"retry","validationCode":"resp, err := http.Get(promURL + \"/api/v1/status/buildinfo\")\nif err != nil || resp.StatusCode != 200 {\n    return fmt.Errorf(\"prometheus flags endpoint unreachable at %s\", promURL)\n}","typeGuard":null,"tryCatchPattern":"err := runutil.Retry(interval, ctx.Done(), validatePrometheusFlags)\nif err != nil {\n    level.Warn(logger).Log(\"msg\", \"prometheus flags validation failed, will retry\", \"err\", err)\n}","preventionTips":["Health-check --prometheus.url from the sidecar network before rollout","Use the in-cluster service DNS name with the correct port 9090","Avoid proxies in front of the /api/v1 paths","Verify Prometheus is up before starting the sidecar (init container or readiness ordering)"],"tags":["prometheus","network","http","startup"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}