thanos-io/thanos · warning

canceled

Error message

canceled

What it means

interrupt() waits for SIGINT/SIGTERM; if instead the cancel channel closes (meaning some other run-group actor failed or the group is shutting down), it returns errors.New("canceled"). This is Thanos's normal way to terminate the signal-waiting actor when shutdown is initiated elsewhere, and it propagates up to cause a clean exit.

Solutions

1. Treat this as a secondary/shutdown error; scan the logs for the earlier root-cause error that closed the cancel channel.
2. If it appears alone, check whether another component was stopped (systemd, k8s liveness kill, docker stop).
3. No code fix needed if shutdown was intentional; fix the root failure otherwise.

// Treat 'canceled' as expected on shutdown
if err := g.Run(); err != nil {
    if errors.Is(errors.Cause(err), context.Canceled) || err.Error() == "canceled" {
        logger.Info("graceful shutdown")
        return nil
    }
    return err
}

When it happens

Trigger: Another run-group actor returns an error or the group is terminated, closing the cancel channel while interrupt() is still blocked waiting on signals; the select then takes the <-cancel branch and returns "canceled".

Common situations: A component (HTTP/gRPC server, store client) failed and triggered group shutdown; or the group's interrupt/ctx cancel fired during graceful shutdown. Often appears alongside the actual root-cause error in logs.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• command failed
• closing connection for

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/1ec31f23c12e4566. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/main.go:183

	}

	if err := g.Run(); err != nil {
		// Use %+v for github.com/pkg/errors error to print with stack.
		level.Error(logger).Log("err", fmt.Sprintf("%+v", errors.Wrapf(err, "%s command failed", cmd)))
		os.Exit(1)
	}
	level.Info(logger).Log("msg", "exiting")
}

func interrupt(logger log.Logger, cancel <-chan struct{}) error {
	c := make(chan os.Signal, 1)
	signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
	select {
	case s := <-c:
		level.Info(logger).Log("msg", "caught signal. Exiting.", "signal", s)
		return nil
	case <-cancel:
		return errors.New("canceled")
	}
}

func reload(logger log.Logger, cancel <-chan struct{}, r chan<- struct{}) error {
	c := make(chan os.Signal, 1)
	signal.Notify(c, syscall.SIGHUP)
	for {
		select {
		case s := <-c:
			level.Info(logger).Log("msg", "caught signal. Reloading.", "signal", s)
			select {
			case r <- struct{}{}:
				level.Info(logger).Log("msg", "reload dispatched.")
			default:
			}
		case <-cancel:
			return errors.New("canceled")
		}

View on GitHub (pinned to 35b8b99117)