thanos-io/thanos · error

command failed

Error message

%s command failed

What it means

Top-level wrapper for the run group: after setup succeeds, Thanos runs all actors via g.Run(); if any actor in the group returns an error, main wraps it with "%s command failed" and logs it with a stack trace, then exits 1. This signals a runtime failure of the command while it was running, not a setup failure.

Solutions

1. Inspect the full %+v stack trace to identify which run-group actor failed.
2. Check that listen addresses (--grpc-address, --http-address) are free: `ss -ltnp` or `lsof -i :<port>`.
3. Verify connectivity to configured store/endpoint peers.
4. Check TLS cert/key file existence and permissions if TLS is enabled.
5. Restart the command after fixing the root cause.

Example fix

// before
thanos query --grpc-address=0.0.0.0:9090  // port already bound
// after
thanos query --grpc-address=0.0.0.0:9091

# Check listen ports are free before start
ss -ltn | grep -E ':(9090|10902)' || echo "ports free"

// systemd unit
Restart=on-failure
RestartSec=5s

When it happens

Trigger: Any actor registered in the run group fails at runtime: gRPC/HTTP listener bind errors, store endpoint connection failures, TLS handshake setup errors, interrupted/canceled context, or component loops returning errors.

Common situations: Port already in use, TLS certificate files unreadable at runtime, downstream store APIs unavailable, or the process received a cancel trigger from another failing actor.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• level is bigger then default set of
• unknown sync strategy
• get compaction levels
• penalty based deduplication needs at least one replica…
• unsupported deduplication func, got

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e003ea844db3f5eb. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/main.go:169

			return interrupt(logger, cancel)
		}, func(error) {
			close(cancel)
		})
	}

	// Listen for reload signals.
	{
		cancel := make(chan struct{})
		g.Add(func() error {
			return reload(logger, cancel, reloadCh)
		}, func(error) {
			close(cancel)
		})
	}

	if err := g.Run(); err != nil {
		// Use %+v for github.com/pkg/errors error to print with stack.
		level.Error(logger).Log("err", fmt.Sprintf("%+v", errors.Wrapf(err, "%s command failed", cmd)))
		os.Exit(1)
	}
	level.Info(logger).Log("msg", "exiting")
}

func interrupt(logger log.Logger, cancel <-chan struct{}) error {
	c := make(chan os.Signal, 1)
	signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
	select {
	case s := <-c:
		level.Info(logger).Log("msg", "caught signal. Exiting.", "signal", s)
		return nil
	case <-cancel:
		return errors.New("canceled")
	}
}

func reload(logger log.Logger, cancel <-chan struct{}, r chan<- struct{}) error {

View on GitHub (pinned to 35b8b99117)