thanos-io/thanos · error

postings offsets for

Error message

postings offsets for %s

What it means

optimizePostingsFetchByDownloadedBytes resolves posting list offsets via the block index header reader (PostingsOffsets) for each postings group it wants to fetch. This error wraps a failure of that index lookup for the given label name, meaning posting offsets for that name could not be read from the block's index (corrupt/partial index, IO error, or unsupported index format).

Solutions

1. Inspect the wrapped underlying error to distinguish decode vs IO failure.
2. Verify the block in object storage is complete (all index files, matching index-header version) and re-upload if truncated.
3. Delete local cached/extracted copies of the block so the gateway re-downloads it.
4. Check that Prometheus index format version is supported by your Thanos build (upgrade if needed).

Example fix

// before: stale index-header left from interrupted upload
// fix by re-syncing/removing the bad block
thanos tools bucket verify --objstore.config-file=bucket.yml --id=<block-id>
thanos tools bucket mark --marker=deletion-mark.json --id=<corrupt-block-id> --objstore.config-file=bucket.yml

// Pre-check before issuing lazy postings query
if err := q.BlockMeta(); err != nil {
	return fmt.Errorf("block unreadable, skip postings optimization: %w", err)
}

rngs, err := r.block.indexHeaderReader.PostingsOffsets(pg.name, pg.addKeys...)
if err != nil {
	if isTransient(err) { // object-store timeout / 5xx
		return nil, false, backoff.Retry(ctx, func() error {
			_, err = r.block.indexHeaderReader.PostingsOffsets(pg.name, pg.addKeys...)
			return err
		})
	}
	logger.Warn("falling back to non-optimized postings fetch", "err", err)
}

When it happens

Trigger: A store-gateway lazy postings fetch (fetchLazyExpandedPostings path) where r.block.indexHeaderReader.PostingsOffsets(name, vals...) fails for a label name — typically because the block's index is truncated/corrupted, the index-header file is missing/stale relative to the index, or a decode error occurs reading the postings offset table.

Common situations: Blocks uploaded incompletely (missing or mismatched .idx versus index-header), object-store flakiness during index read, Thanos version reading blocks written by an incompatible Prometheus index format, corrupted local cache of block index files.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• get all postings
• read meta file
• get postings
• critical error detected
• create meta fetcher

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a4f8a3b246e5b740. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/lazy_postings.go:63

	seriesMaxSize int64,
	seriesMatchRatio float64,
	postingGroupMaxKeySeriesRatio float64,
	lazyExpandedPostingSizeBytes prometheus.Counter,
	lazyExpandedPostingGroupsByReason *prometheus.CounterVec,
) ([]*postingGroup, bool, error) {
	if len(postingGroups) <= 1 {
		return postingGroups, false, nil
	}
	// Collect posting cardinality of each posting group.
	for _, pg := range postingGroups {
		// A posting group can have either add keys or remove keys but not both the same time.
		vals := pg.addKeys
		if len(pg.removeKeys) > 0 {
			vals = pg.removeKeys
		}
		rngs, err := r.block.indexHeaderReader.PostingsOffsets(pg.name, vals...)
		if err != nil {
			return nil, false, errors.Wrapf(err, "postings offsets for %s", pg.name)
		}

		existentKeys := 0
		for _, rng := range rngs {
			if rng == indexheader.NotFoundRange {
				continue
			}
			if rng.End <= rng.Start {
				level.Error(r.logger).Log("msg", "invalid index range, fallback to non lazy posting optimization")
				return postingGroups, false, nil
			}
			existentKeys++
			// Each range starts from the #entries field which is 4 bytes.
			// Need to subtract it when calculating number of postings.
			// https://github.com/prometheus/prometheus/blob/v2.46.0/tsdb/docs/format/index.md.
			pg.cardinality += (rng.End - rng.Start - 4) / 4
		}
		pg.existentKeys = existentKeys

View on GitHub (pinned to 35b8b99117)