{"record":{"id":"82a8421b1b4f0b7c","repo":"thanos-io/thanos","slug":"aborting-as-no-external-labels-found-after-waiting","errorCode":null,"errorMessage":"aborting as no external labels found after waiting %s","messagePattern":"aborting as no external labels found after waiting (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/sidecar.go","lineNumber":457,"sourceCode":"\t\tif err := promclient.IsWALDirAccessible(conf.tsdb.path); err != nil {\n\t\t\tlevel.Error(logger).Log(\"err\", err)\n\t\t}\n\n\t\tctx, cancel := context.WithCancel(context.Background())\n\t\tg.Add(func() error {\n\t\t\tdefer runutil.CloseWithLogOnErr(logger, bkt, \"bucket client\")\n\n\t\t\tpromReadyTimeout := conf.prometheus.readyTimeout\n\t\t\textLabelsCtx, cancel := context.WithTimeout(ctx, promReadyTimeout)\n\t\t\tdefer cancel()\n\n\t\t\tif err := runutil.Retry(2*time.Second, extLabelsCtx.Done(), func() error {\n\t\t\t\tif m.Labels().Len() == 0 {\n\t\t\t\t\treturn errors.New(\"not uploading as no external labels are configured yet - is Prometheus healthy/reachable?\")\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t}); err != nil {\n\t\t\t\treturn errors.Wrapf(err, \"aborting as no external labels found after waiting %s\", promReadyTimeout)\n\t\t\t}\n\n\t\t\tdataDir, err := os.OpenRoot(conf.tsdb.path)\n\t\t\tif err != nil {\n\t\t\t\treturn errors.Wrap(err, \"opening tsdb directory\")\n\t\t\t}\n\t\t\tdefer runutil.CloseWithLogOnErr(logger, dataDir, \"tsdb directory\")\n\n\t\t\ts := shipper.New(\n\t\t\t\tbkt,\n\t\t\t\tdataDir,\n\t\t\t\tshipper.WithLogger(logger),\n\t\t\t\tshipper.WithRegisterer(reg),\n\t\t\t\tshipper.WithSource(metadata.SidecarSource),\n\t\t\t\tshipper.WithHashFunc(metadata.HashFunc(conf.shipper.hashFunc)),\n\t\t\t\tshipper.WithMetaFileName(conf.shipper.metaFileName),\n\t\t\t\tshipper.WithLabels(m.Labels),\n\t\t\t\tshipper.WithUploadCompacted(conf.shipper.uploadCompacted),","sourceCodeStart":439,"sourceCodeEnd":475,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/sidecar.go#L439-L475","documentation":"This wrap is the terminal failure of the external-labels wait loop: after runutil.Retry gives up (either the per-label-check error kept firing or the extLabelsCtx timeout expired), the sidecar aborts startup with this message. It means Prometheus external labels never appeared within promReadyTimeout, so block shipping cannot proceed safely.","triggerScenarios":"The sidecar ran the 2-second-interval label check for the full promReadyTimeout (default 10m) and m.Labels().Len() stayed 0, or ctx was cancelled during the retry; the retry error is then wrapped as 'aborting as no external labels found after waiting <duration>'.","commonSituations":"Prometheus started without external_labels and stayed that way for the whole timeout; the sidecar is pointed at the wrong Prometheus; the whole Thanos process was shut down (ctx done) while waiting.","solutions":["Configure external_labels in prometheus.yml and reload, then restart the sidecar.","Fix --prometheus.url if it targets the wrong instance.","Raise the ready timeout flag if startup ordering is the issue.","If 'context canceled' is the wrapped cause, check why the parent ctx was cancelled (shutdown signal)."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if !waitForExternalLabels(promURL, timeout) { failFast(\"external labels missing\") }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Provision external_labels before first Prometheus start, not after.","Set --prometheus.ready-timeout above your Prometheus startup time.","Alert on sidecar restart loops with this message."],"tags":["thanos-sidecar","external-labels","startup-failure","timeout"],"backgroundTag":"missing-required-config","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}