thanos-io/thanos · error

iterate series for block

Error message

iterate series for block %s

What it means

During LabelNames, the store gateway fans out to per-block clients (StoreAPI series streams) and reads responses with blockClient.Recv(). Non-EOF receive errors for a block are wrapped with the block's ULID — the underlying per-block series RPC failed mid-stream.

Solutions

1. Retry the LabelNames request; transient Unavailable errors usually clear.
2. Check logs/health of the store instance serving that block ULID.
3. Increase the request deadline or limit concurrency if the error is DeadlineExceeded.
4. Verify block availability; remove or re-sync stale block metadata if the block no longer exists.

if st, ok := status.FromError(err); ok && st.Code() == codes.Unavailable {
    time.Sleep(backoff)
    // retry the LabelNames request
}

When it happens

Trigger: blockClient.Recv() returns a transport/gRPC error (Unavailable, DeadlineExceeded, Canceled) while iterating series for a block: remote store crashed, network interruption, per-request deadline hit.

Common situations: Store gateway to sidecar/remote store network blips during heavy queries; block served by an instance that went away; deadline too short for huge blocks.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• receiving metric metadata from metadata client
• sending rules warning to server
• Unknown
• receiving tsdb statistics from status client
• fetch postings for block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b35b4591f723f7b4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:1994

				if err := blockClient.ExpandPostings(
					sortedReqSeriesMatchersNoExtLabels,
					seriesLimiter,
				); err != nil {
					return err
				}

				// Extract label names from all series. Many label names will be the same, so we need to deduplicate them.
				// Note that label names will already include external labels (passed to blockSeries), so we don't need
				// to add them again.
				labelNames := map[string]struct{}{}
				for {
					ls, err := blockClient.Recv()
					if err == io.EOF {
						break
					}
					if err != nil {
						return errors.Wrapf(err, "iterate series for block %s", b.meta.ULID)
					}

					if ls.GetWarning() != "" {
						return errors.Wrapf(errors.New(ls.GetWarning()), "iterate series for block %s", b.meta.ULID)
					}
					if ls.GetSeries() == nil {
						continue
					}
					for _, l := range ls.GetSeries().Labels {
						labelNames[l.Name] = struct{}{}
					}
				}

				result = make([]string, 0, len(labelNames))
				for n := range labelNames {
					result = append(result, n)
				}
				sort.Strings(result)

View on GitHub (pinned to 35b8b99117)