thanos-io/thanos · error

index header label values for block

Error message

index header label values for block %s

What it means

For each queried block, BucketStore reads label values from the block's in-memory index header (indexHeaderReader.LabelValues). A failure reading/decoding the cached index-header file for that block is wrapped with the block ULID and aborts the LabelValues computation.

Solutions

1. Clear the index-header cache directory for the affected block so it is re-fetched from object storage
2. Verify the object-store block index is intact (the underlying err names the real cause)
3. Restart the store gateway to rebuild the cache if entries are corrupt
4. Check disk space/permissions on the cache mount; upgrade thanos if the format changed across versions

Example fix

// before (cache entry corrupt, retry keeps failing)
// rm -rf <cachedir>/index-header/<block-ULID> then restart store gateway
// after: code-side handling
res, err := indexr.block.indexHeaderReader.LabelValues(req.Label)
if err != nil {
    b.dropIndexHeaderCache() // force re-download
    return errors.Wrapf(err, "index header label values for block %s", b.meta.ULID)
}

if _, err := os.Stat(cacheIndexPath); err != nil || cacheCorrupt(cacheIndexPath) { invalidateCacheEntry() }

res, err := indexr.block.indexHeaderReader.LabelValues(req.Label)
if err != nil {
    return errors.Wrapf(err, "index header label values for block %s", b.meta.ULID)
}

When it happens

Trigger: The block's index header cache file is corrupt, was written by an incompatible thanos version, or was deleted/corrupted in the cache directory while the block is being queried.

Common situations: Upgrades where the on-disk index-header format changed; truncated downloads into the index-header cache; disk full or race during cache write; block deleted from bucket mid-query with stale cache.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• invalid magic number
• write index header
• generate index header
• write index header
• create index header reader

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/3f153b7cbabf2c4b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:2155

			reqSeriesMatchersNoExtLabels = append(reqSeriesMatchersNoExtLabels, m)
		}

		sortedReqSeriesMatchersNoExtLabels := newSortedMatchers(reqSeriesMatchersNoExtLabels)

		resHints.AddQueriedBlock(b.meta.ULID)

		blockLogger := log.With(logger, "block", b.meta.ULID)
		indexr := b.indexReader(blockLogger)

		g.Go(func() error {
			defer runutil.CloseWithLogOnErr(blockLogger, indexr, "label values")

			var result []string
			if len(reqSeriesMatchersNoExtLabels) == 0 {
				// Do it via index reader to have pending reader registered correctly.
				res, err := indexr.block.indexHeaderReader.LabelValues(req.Label)
				if err != nil {
					return errors.Wrapf(err, "index header label values for block %s", b.meta.ULID)
				}

				// Add the external label value as well.
				if extLabelValue := b.extLset.Get(req.Label); extLabelValue != "" {
					res = strutil.MergeSlices(int(req.Limit), res, []string{extLabelValue})
				}
				result = res
			} else {
				seriesReq := &storepb.SeriesRequest{
					MinTime:              req.Start,
					MaxTime:              req.End,
					SkipChunks:           true,
					WithoutReplicaLabels: req.WithoutReplicaLabels,
				}
				blockClient := newBlockSeriesClient(
					gctx,
					blockLogger,
					b,

View on GitHub (pinned to 35b8b99117)