thanos-io/thanos · critical

tracing failed

Error message

tracing failed

What it means

Raised at Thanos binary startup when initializing the distributed tracer from the provided tracing config YAML fails. The error is printed to stderr and the process exits immediately with status 1, since tracing is required before serving. Common clients: Jaeger. The underlying cause (bad config, exporter connection setup) is wrapped.

Solutions

1. Validate the tracing config YAML against the tracing.Config schema (type: jaeger, service_name, etc.) for your Thanos version.
2. Start with an empty/no --tracing.config to confirm the binary runs (NoopTracer), then add tracing incrementally.
3. Check Jaeger endpoint/agent address reachability and fix host:port or sampler params in the config.
4. Remove deprecated tracing fields per the Thanos release notes if you upgraded and the schema changed.

Example fix

// before
// tracing:
//   type: jaeger
//   config:
//     endpoint: bad-host:14268
// after
// tracing:
//   type: jaeger
//   service_name: thanos-query
//   config:
//     endpoint: jaeger-collector.monitoring.svc:14268

var tc tracing.Config
if err := yaml.Unmarshal(confContentYaml, &tc); err != nil { return fmt.Errorf("tracing config invalid: %w", err) }
if tc.Type == "" && len(confContentYaml) > 0 { return errors.New("tracing.type is required when tracing config is provided") }

tracer, closer, err = client.NewTracer(ctx, logger, metrics, confContentYaml)
if err != nil {
    logger.Error("tracing failed, continuing without tracing", "err", err)
    tracer, closer = client.NoopTracer(), nil // optional: degrade instead of exit
}

When it happens

Trigger: --tracing.config (or tracing.config-file) content is non-empty and client.NewTracer fails: invalid YAML, unknown tracer type, missing required fields (e.g., jaeger endpoint), or failure connecting/configuring the exporter.

Common situations: Operator supplies tracing config referencing jaeger with unreachable agent/collector — depending on client, this can fail eagerly; typo in config keys; deprecated tracing config schema after Thanos upgrade; empty config correctly falls back to NoopTracer so only non-empty bad config triggers this.

Related errors

• unable to load config file
• unable to create config reloader
• unable to load config initially
• query configuration
• invalid alert source template

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/66600709ad8679a1. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/main.go:115

	{
		var (
			ctx             = context.Background()
			closer          io.Closer
			confContentYaml []byte
		)

		confContentYaml, err = tracingConfig.Content()
		if err != nil {
			level.Error(logger).Log("msg", "getting tracing config failed", "err", err)
			os.Exit(1)
		}

		if len(confContentYaml) == 0 {
			tracer = client.NoopTracer()
		} else {
			tracer, closer, err = client.NewTracer(ctx, logger, metrics, confContentYaml)
			if err != nil {
				fmt.Fprintln(os.Stderr, errors.Wrapf(err, "tracing failed"))
				os.Exit(1)
			}
		}

		// This is bad, but Prometheus does not support any other tracer injections than just global one.
		// TODO(bplotka): Work with basictracer to handle gracefully tracker mismatches, and also with Prometheus to allow
		// tracer injection.
		opentracing.SetGlobalTracer(tracer)

		ctx, cancel := context.WithCancel(ctx)
		g.Add(func() error {
			<-ctx.Done()
			return ctx.Err()
		}, func(error) {
			if closer != nil {
				if err := closer.Close(); err != nil {
					level.Warn(logger).Log("msg", "closing tracer failed", "err", err)
				}

View on GitHub (pinned to 35b8b99117)