thanos-io/thanos · error

aborting as no external labels found after waiting

Error message

aborting as no external labels found after waiting %s

What it means

This wrap is the terminal failure of the external-labels wait loop: after runutil.Retry gives up (either the per-label-check error kept firing or the extLabelsCtx timeout expired), the sidecar aborts startup with this message. It means Prometheus external labels never appeared within promReadyTimeout, so block shipping cannot proceed safely.

Solutions

1. Configure external_labels in prometheus.yml and reload, then restart the sidecar.
2. Fix --prometheus.url if it targets the wrong instance.
3. Raise the ready timeout flag if startup ordering is the issue.
4. If 'context canceled' is the wrapped cause, check why the parent ctx was cancelled (shutdown signal).

if !waitForExternalLabels(promURL, timeout) { failFast("external labels missing") }

When it happens

Trigger: The sidecar ran the 2-second-interval label check for the full promReadyTimeout (default 10m) and m.Labels().Len() stayed 0, or ctx was cancelled during the retry; the retry error is then wrapped as 'aborting as no external labels found after waiting <duration>'.

Common situations: Prometheus started without external_labels and stayed that way for the whole timeout; the sidecar is pointed at the wrong Prometheus; the whole Thanos process was shut down (ctx done) while waiting.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• not uploading as no external labels are configured yet - is…
• failed to get tsdb status from prometheus
• opening tsdb directory
• found that Prometheus and Thanos use different paths for…
• no external labels configured, uniquely identifying…

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/82a8421b1b4f0b7c. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/sidecar.go:457

		if err := promclient.IsWALDirAccessible(conf.tsdb.path); err != nil {
			level.Error(logger).Log("err", err)
		}

		ctx, cancel := context.WithCancel(context.Background())
		g.Add(func() error {
			defer runutil.CloseWithLogOnErr(logger, bkt, "bucket client")

			promReadyTimeout := conf.prometheus.readyTimeout
			extLabelsCtx, cancel := context.WithTimeout(ctx, promReadyTimeout)
			defer cancel()

			if err := runutil.Retry(2*time.Second, extLabelsCtx.Done(), func() error {
				if m.Labels().Len() == 0 {
					return errors.New("not uploading as no external labels are configured yet - is Prometheus healthy/reachable?")
				}
				return nil
			}); err != nil {
				return errors.Wrapf(err, "aborting as no external labels found after waiting %s", promReadyTimeout)
			}

			dataDir, err := os.OpenRoot(conf.tsdb.path)
			if err != nil {
				return errors.Wrap(err, "opening tsdb directory")
			}
			defer runutil.CloseWithLogOnErr(logger, dataDir, "tsdb directory")

			s := shipper.New(
				bkt,
				dataDir,
				shipper.WithLogger(logger),
				shipper.WithRegisterer(reg),
				shipper.WithSource(metadata.SidecarSource),
				shipper.WithHashFunc(metadata.HashFunc(conf.shipper.hashFunc)),
				shipper.WithMetaFileName(conf.shipper.metaFileName),
				shipper.WithLabels(m.Labels),
				shipper.WithUploadCompacted(conf.shipper.uploadCompacted),

View on GitHub (pinned to 35b8b99117)