{"record":{"id":"3819b2eecde45209","repo":"thanos-io/thanos","slug":"initial-external-labels-query","errorCode":null,"errorMessage":"initial external labels query","messagePattern":"initial external labels query","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/sidecar.go","lineNumber":266,"sourceCode":"\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tif err := m.UpdateLabels(iterCtx); err != nil {\n\t\t\t\t\tlevel.Warn(logger).Log(\n\t\t\t\t\t\t\"msg\", \"failed to fetch initial external labels. Is Prometheus running? Retrying\",\n\t\t\t\t\t\t\"err\", err,\n\t\t\t\t\t)\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\n\t\t\t\t\t\"msg\", \"successfully loaded prometheus external labels\",\n\t\t\t\t\t\"external_labels\", m.Labels().String(),\n\t\t\t\t)\n\t\t\t\treturn nil\n\t\t\t})\n\t\t\tif err != nil {\n\t\t\t\treturn errors.Wrap(err, \"initial external labels query\")\n\t\t\t}\n\n\t\t\tif m.Labels().Len() == 0 {\n\t\t\t\treturn errors.New(\"no external labels configured on Prometheus server, uniquely identifying external labels must be configured; see https://thanos.io/tip/thanos/storage.md#external-labels for details.\")\n\t\t\t}\n\t\t\tpromUp.Set(1)\n\t\t\tstatusProber.Ready()\n\n\t\t\tclose(readyToStartGRPC)\n\n\t\t\t// Periodically query the Prometheus config. We use this as a heartbeat as well as for updating\n\t\t\t// the external labels we apply.\n\t\t\treturn runutil.Repeat(conf.prometheus.getConfigInterval, ctx.Done(), func() error {\n\t\t\t\titerCtx, iterCancel := context.WithTimeout(context.Background(), conf.prometheus.getConfigTimeout)\n\t\t\t\tdefer iterCancel()\n\t\t\t\tif err := m.UpdateTimestamps(iterCtx); err != nil {\n\t\t\t\t\tlevel.Warn(logger).Log(\"msg\", \"updating timestamps failed\", \"err\", err)\n\t\t\t\t\tpromUp.Set(0)","sourceCodeStart":248,"sourceCodeEnd":284,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/sidecar.go#L248-L284","documentation":"The blocking initial query of Prometheus external labels (GET /api/v1/status/config) failed. The sidecar must fetch external labels once before joining the store/gossip cluster; any client error from that first call is wrapped as 'initial external labels query'.","triggerScenarios":"errors.Wrap after the retried label fetch in runSidecar: promClient.Labels(ctx) failed — --prometheus.url unreachable, malformed Prometheus config returned by the API, request timeout, or Prometheus returned non-2xx during the loop.","commonSituations":"Prometheus still starting when the sidecar boots and retries exhaust; network policy blocking sidecar→Prometheus; Prometheus config endpoint returning an unexpected payload.","solutions":["Verify curl http://<prometheus-url>/api/v1/status/config works from the sidecar","Fix --prometheus.url / DNS / network policies","Wait for Prometheus readiness (or add readiness ordering) and restart the sidecar","Increase --prometheus.get-config-interval/timeout so retries span a slow Prometheus start"],"exampleFix":"// before\n--prometheus.get-config-timeout=2s   # too short\n// after\n--prometheus.get-config-timeout=30s --prometheus.get-config-interval=5s","handlingStrategy":"retry","validationCode":"resp, err := http.Get(promURL + \"/api/v1/status/config\")\nif err != nil || resp.StatusCode != 200 {\n    return fmt.Errorf(\"prometheus config endpoint unreachable\")\n}","typeGuard":null,"tryCatchPattern":"err := runutil.Retry(interval, ctx.Done(), func() error {\n    return promClient.Labels(ctx)\n})\nif err != nil {\n    return errors.Wrap(err, \"initial external labels query\")\n}","preventionTips":["Order startup so Prometheus is ready before the sidecar","Tune get-config-interval/timeout for slow starts","Test /api/v1/status/config from inside the cluster","Keep Prometheus config minimal during first boot to avoid API errors"],"tags":["prometheus","http","labels","startup"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}