thanos-io/thanos · warning

Not ready

Error message

Not ready

What it means

The rule component's Rules/Store gRPC service builds its TSDB store lazily; the info provider callback returns errors.New("Not ready") when httpProbe.IsReady() is false, i.e. the component has finished registering but is not yet fully initialized (TSDB/store not yet set up). Clients querying the store endpoint during this window receive this error.

Solutions

  1. Retry with backoff on the client side; the error clears once initialization completes.
  2. Gate traffic on the component's HTTP readiness probe (-/-ready) before sending store requests.
  3. Check startup logs for a hung initialization step (e.g. bucket verification, TSDB open) if the state persists.
  4. Configure scrape/discovery refresh intervals to avoid hammering during startup.
Defensive patterns

Strategy: retry

Validate before calling

// gate traffic on readiness first:
until curl -sf http://rule:10902/-/ready; do sleep 1; done

Try / catch

// retry with backoff
for i := 0; i < 10; i++ {
  resp, err := client.Info(ctx, req)
  if err == nil || !strings.Contains(err.Error(), "Not ready") { return resp, err }
  time.Sleep(backoff(i))
}

Prevention

When it happens

Trigger: A store/rules gRPC request (e.g. Info or Series) hits the rule component between process start and readiness: the info provider func runs while initialization (TSDB open, shipping setup) is still in progress.

Common situations: Prometheus/Querier scraping store API info immediately after the pod starts; load balancer sending traffic before the readiness probe flips; service discovery registering the endpoint before IsReady becomes true.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a0739fe1fe9809a2. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:780

	if tsdbDB != nil {
		tsdbStore := store.NewTSDBStore(logger, tsdbDB, component.Rule, conf.lset)
		infoOptions = append(
			infoOptions,
			info.WithLabelSetFunc(func() []labelpb.ZLabelSet {
				return tsdbStore.LabelSet()
			}),
			info.WithStoreInfoFunc(func() (*infopb.StoreInfo, error) {
				if httpProbe.IsReady() {
					mint, maxt := tsdbStore.TimeRange()
					return &infopb.StoreInfo{
						MinTime:                      mint,
						MaxTime:                      maxt,
						SupportsSharding:             true,
						SupportsWithoutReplicaLabels: true,
						TsdbInfos:                    tsdbStore.TSDBInfos(),
					}, nil
				}
				return nil, errors.New("Not ready")
			}),
			info.WithStatusInfoFunc(),
		)
		storeServer := store.NewLimitedStoreServer(store.NewInstrumentedStoreServer(reg, tsdbStore), reg, conf.storeRateLimits)
		options = append(options, grpcserver.WithServer(store.RegisterStoreServer(storeServer, logger)))

		// Add Status server for TSDB statistics
		statusSrv := status.NewServer(
			component.Rule.String(),
			status.WithTSDBStatisticsGetter(
				status.TSDBStatisticsGetterFunc(func(limit int, matchers []storepb.LabelMatcher) (map[string]tsdb.Stats, error) {
					if !httpProbe.IsReady() {
						return nil, errors.New("not ready")
					}

					// Check if external labels match the provided matchers.
					extLabels := conf.lset
					promMatchers, err := storepb.MatchersToPromMatchers(matchers...)

View on GitHub (pinned to 35b8b99117)