thanos-io/thanos · error

flushing storage

Error message

flushing storage

What it means

After a hashring change (and non-Ketama algorithm or first initialization), the TSDB head must be flushed/compacted so blocks can be re-sharded; this error is surfaced when the flush step fails, leaving the server not-ready.

Solutions

1. Check disk space for compaction output
2. Inspect TSDB errors in logs
3. Retry after fixing storage issues

When it happens

Trigger: Thrown at cmd/thanos/receive.go:769 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/590cfb6f35e5ada6. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/receive.go:769

				}

				// When using Ketama as the hashring algorithm, there is no need to flush the TSDB head.
				// If new receivers were added to the hashring, existing receivers will not need to
				// ingest additional series.
				// If receivers are removed from the hashring, existing receivers will only need
				// to ingest a subset of the series that were assigned to the removed receivers.
				// As a result, changing the hashring produces no churn, hence no need to force
				// head compaction and upload.
				flushHead := !initialized || hashringAlgorithm != receive.AlgorithmKetama
				if flushHead {
					msg := "hashring has changed; server is not ready to receive requests"
					statusProber.NotReady(errors.New(msg))
					level.Info(logger).Log("msg", msg)

					level.Info(logger).Log("msg", "updating storage")
					dbUpdatesStarted.Inc()
					if err := dbs.Flush(); err != nil {
						return errors.Wrap(err, "flushing storage")
					}
					if err := dbs.Open(); err != nil {
						return errors.Wrap(err, "opening storage")
					}
					if upload {
						uploadC <- struct{}{}
						<-uploadDone
					}
					dbUpdatesCompleted.Inc()
					statusProber.Ready()
					level.Info(logger).Log("msg", "storage started, and server is ready to receive requests")
					dbUpdatesCompleted.Inc()
				}
				initialized = true
			}
		}
	}, func(err error) {
		close(cancel)

View on GitHub (pinned to 35b8b99117)