thanos-io/thanos · error

hashring has changed; server is not ready to receive…

Error message

hashring has changed; server is not ready to receive requests

What it means

Not an error object but a readiness message: when the hashring changes, the server is marked not-ready and this message is logged/probed while head flushing and reopening complete. It fires on every hashring update (unless Ketama with no flush needed).

Solutions

1. Wait for flush/reopen to complete; readiness restores automatically
2. During frequent hashring churn consider Ketama to reduce flushes
3. Scale replication changes gradually

When it happens

Trigger: Thrown at cmd/thanos/receive.go:763 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4022d11e54dff79e. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/receive.go:763

			select {
			case <-cancel:
				return nil
			case _, ok := <-hashringChangedChan:
				if !ok {
					return nil
				}

				// When using Ketama as the hashring algorithm, there is no need to flush the TSDB head.
				// If new receivers were added to the hashring, existing receivers will not need to
				// ingest additional series.
				// If receivers are removed from the hashring, existing receivers will only need
				// to ingest a subset of the series that were assigned to the removed receivers.
				// As a result, changing the hashring produces no churn, hence no need to force
				// head compaction and upload.
				flushHead := !initialized || hashringAlgorithm != receive.AlgorithmKetama
				if flushHead {
					msg := "hashring has changed; server is not ready to receive requests"
					statusProber.NotReady(errors.New(msg))
					level.Info(logger).Log("msg", msg)

					level.Info(logger).Log("msg", "updating storage")
					dbUpdatesStarted.Inc()
					if err := dbs.Flush(); err != nil {
						return errors.Wrap(err, "flushing storage")
					}
					if err := dbs.Open(); err != nil {
						return errors.Wrap(err, "opening storage")
					}
					if upload {
						uploadC <- struct{}{}
						<-uploadDone
					}
					dbUpdatesCompleted.Inc()
					statusProber.Ready()
					level.Info(logger).Log("msg", "storage started, and server is ready to receive requests")
					dbUpdatesCompleted.Inc()

View on GitHub (pinned to 35b8b99117)