thanos-io/thanos · error

fetch label values from store

Error message

fetch label values from store %s

What it means

A fan-out LabelValues RPC to one store in the StoreAPI fan-out failed (connection error, store-side error, or cancelled errgroup context). The message identifies which store could not serve label values; the error is aggregated per store and turned into a warning when partial responses are allowed.

Solutions

1. Inspect the failing store's health, logs and gRPC endpoint
2. Verify service discovery is returning a reachable address for the store
3. Retry the query after the store recovers
4. Use partial responses to get results from the remaining stores, then re-query later

When it happens

Trigger: Thrown at pkg/store/proxy.go:569 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7b9a80c8301dd12b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/proxy.go:569

		PartialResponseStrategy: originalRequest.PartialResponseStrategy,
		Start:                   originalRequest.Start,
		End:                     originalRequest.End,
		Matchers:                append(storeMatchers, MatchersForLabelSets(storeLabelSets)...),
		WithoutReplicaLabels:    originalRequest.WithoutReplicaLabels,
		Limit:                   originalRequest.Limit,
	}

	var (
		warnings []string
		all      [][]string
		mtx      sync.Mutex
		g, gctx  = errgroup.WithContext(ctx)
	)
	for _, st := range stores {
		g.Go(func() error {
			resp, err := st.LabelValues(gctx, r)
			if err != nil {
				err = errors.Wrapf(err, "fetch label values from store %s", st)
				if r.PartialResponseDisabled || r.PartialResponseStrategy == storepb.PartialResponseStrategy_ABORT {
					return err
				}

				mtx.Lock()
				warnings = append(warnings, err.Error())
				mtx.Unlock()
				return nil
			}

			mtx.Lock()
			warnings = append(warnings, resp.Warnings...)
			all = append(all, resp.Values)
			mtx.Unlock()

			return nil
		})
	}

View on GitHub (pinned to 35b8b99117)