{"record":{"id":"47917a2cf4ee4df8","repo":"thanos-io/thanos","slug":"no-external-labels-configured-on-prometheus-server","errorCode":null,"errorMessage":"no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.","messagePattern":"no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos\\.io/tip/thanos/storage\\.md#external-labels for details\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/thanos/sidecar.go","lineNumber":270,"sourceCode":"\t\t\t\t\tlevel.Warn(logger).Log(\n\t\t\t\t\t\t\"msg\", \"failed to fetch initial external labels. Is Prometheus running? Retrying\",\n\t\t\t\t\t\t\"err\", err,\n\t\t\t\t\t)\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\n\t\t\t\t\t\"msg\", \"successfully loaded prometheus external labels\",\n\t\t\t\t\t\"external_labels\", m.Labels().String(),\n\t\t\t\t)\n\t\t\t\treturn nil\n\t\t\t})\n\t\t\tif err != nil {\n\t\t\t\treturn errors.Wrap(err, \"initial external labels query\")\n\t\t\t}\n\n\t\t\tif m.Labels().Len() == 0 {\n\t\t\t\treturn errors.New(\"no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.\")\n\t\t\t}\n\t\t\tpromUp.Set(1)\n\t\t\tstatusProber.Ready()\n\n\t\t\tclose(readyToStartGRPC)\n\n\t\t\t// Periodically query the Prometheus config. We use this as a heartbeat as well as for updating\n\t\t\t// the external labels we apply.\n\t\t\treturn runutil.Repeat(conf.prometheus.getConfigInterval, ctx.Done(), func() error {\n\t\t\t\titerCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)\n\t\t\t\tdefer iterCancel()\n\t\t\t\tif err := m.UpdateTimestamps(iterCtx); err != nil {\n\t\t\t\t\tlevel.Warn(logger).Log(\"msg\", \"updating timestamps failed\", \"err\", err)\n\t\t\t\t\tpromUp.Set(0)\n\t\t\t\t\tstatusProber.NotReady(err)\n\t\t\t\t\treturn nil\n\t\t\t\t}\n","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/sidecar.go#L252-L288","documentation":"The sidecar requires Prometheus to have uniquely identifying external labels configured; if the fetched external labels set is empty, it returns this hard error and refuses to start, because blocks uploaded by this sidecar would otherwise be indistinguishable from other Prometheus instances.","triggerScenarios":"errors.New in runSidecar right after the successful initial external labels query: m.Labels().Len() == 0, i.e. the Prometheus server has no external_labels (or only labels Thanos drops, e.g. replica/promise labels handling differs across versions).","commonSituations":"Fresh Prometheus deployment without external_labels; global.external_labels removed in a Helm values refactor; users pointing the sidecar at a Prometheus not meant for Thanos.","solutions":["Add external_labels to the Prometheus configuration, e.g. external_labels: {cluster: <name>, replica: \"0\"}, then reload Prometheus and restart the sidecar","Ensure external_labels are set globally in prometheus.yml, not only on scrape configs","If this Prometheus is not meant for Thanos, stop pointing the sidecar at it","Verify labels were actually applied: curl /api/v1/status/config and check the global external_labels section"],"exampleFix":"// before\n# prometheus.yml\nglobal:\n  scrape_interval: 15s\n// after\n# prometheus.yml\nglobal:\n  scrape_interval: 15s\n  external_labels:\n    cluster: eu1\n    replica: \"0\"","handlingStrategy":"validation","validationCode":"cfg := loadPrometheusConfig()\nif len(cfg.Global.ExternalLabels.Map()) == 0 {\n    return errors.New(\"prometheus external_labels must be configured for thanos sidecar\")\n}","typeGuard":null,"tryCatchPattern":"if err := run(); err != nil {\n    if strings.Contains(err.Error(), \"no external labels\") {\n        level.Error(logger).Log(\"msg\", \"fix prometheus.yml: set global external_labels (cluster, replica)\")\n    }\n    return err\n}","preventionTips":["Always set external_labels (cluster + replica) in prometheus.yml global section","Lint prometheus.yml in CI to assert external_labels present when sidecar is deployed","Keep labels stable across restarts to avoid duplicate block uploads","Document the requirement in deployment templates/Helm values"],"tags":["prometheus","config","labels","thanos"],"backgroundTag":"missing-required-config","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}