thanos-io/thanos · error

get postings

Error message

get postings

What it means

fetchAndExpandPostingGroups in Thanos' lazy postings path fetches posting groups from the index and wraps any upstream fetch failure with "get postings". It indicates the underlying index (e.g. a bucket store block) failed to return postings for one of the query's matcher groups. The deferred loop closes all open resources before returning, so the failure is cleanup-safe.

Solutions

1. Inspect the wrapped inner error (%v) to find the root cause — network, object storage, or cancellation.
2. Retry the query; transient storage/network failures are the most common cause.
3. Verify block indexes in the bucket are intact and not deleted mid-query (check store-gateway blocks sync).
4. Increase query/store timeouts if the context deadline was the trigger.

Example fix

// before
res, err := store.Series(ctx, hints, matchers...)
// after
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
res, err := store.Series(ctx, hints, matchers...)
if err != nil {
  if errors.Is(err, context.DeadlineExceeded) { /* retry with longer timeout */ }
}

if err := ctx.Err(); err != nil { return err } // ensure ctx alive before query
if len(matchers) == 0 { return errors.New("need at least one matcher") }

if err != nil {
  if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
    // do not retry; caller gave up
  } else if strings.Contains(err.Error(), "get postings") {
    // retry with backoff against object storage
  }
}

When it happens

Trigger: Calling fetchLazyExpandedPostings during a StoreAPI Series request when an underlying posting fetch fails (corrupt/missing index, network error fetching index from object storage, or a cancelled/expired context mid-fetch).

Common situations: Object storage throttling or timeouts when Thanos Store Gateway fetches index headers; deleted blocks whose index disappeared between listing and fetch; query cancellation when the request context deadline is exceeded.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• get all postings
• get series
• symbols
• get appender
• postings offsets for

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8f26065d923aa863. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/lazy_postings.go:306

			}
		}

		i++
	}

	return keys, lazyMatchers
}

func fetchAndExpandPostingGroups(ctx context.Context, r *bucketIndexReader, postingGroups []*postingGroup, bytesLimiter BytesLimiter, tenant string) ([]storage.SeriesRef, []*labels.Matcher, error) {
	keys, lazyMatchers := keysToFetchFromPostingGroups(postingGroups)
	fetchedPostings, closeFns, err := r.fetchPostings(ctx, keys, bytesLimiter, tenant)
	defer func() {
		for _, closeFn := range closeFns {
			closeFn()
		}
	}()
	if err != nil {
		return nil, nil, errors.Wrap(err, "get postings")
	}

	result := mergeFetchedPostings(ctx, fetchedPostings, postingGroups)
	if err := ctx.Err(); err != nil {
		return nil, nil, err
	}
	ps, err := ExpandPostingsWithContext(ctx, result)
	r.postings = ps
	if err != nil {
		return nil, nil, errors.Wrap(err, "expand")
	}
	return ps, lazyMatchers, nil
}

func mergeFetchedPostings(ctx context.Context, fetchedPostings []index.Postings, postingGroups []*postingGroup) index.Postings {
	// Get "add" and "remove" postings from groups. We iterate over postingGroups and their keys
	// again, and this is exactly the same order as before (when building the groups), so we can simply
	// use one incrementing index to fetch postings from returned slice.

View on GitHub (pinned to 35b8b99117)