thanos-io/thanos · error

metric label values

Error message

metric label values

What it means

GatherIndexHealthStats calls r.LabelValues(ctx, "__name__", nil) to count metric label values. Failure of that postings lookup is wrapped as "metric label values". Like the label-names failure, it points at a damaged or unreadable index postings/symbols table for the __name__ label.

Solutions

1. Restore/re-download the block from object storage
2. Verify the block with VerifyIndex to confirm index corruption
3. Bump the verification context deadline
4. Remove the unreadable block so it can be re-synced

Example fix

// before
stats, err := block.GatherIndexHealthStats(ctx, bdir) // metric label values: unexpected EOF
// after
f, err := os.Open(filepath.Join(bdir, "index"))
if err != nil { return err }
fi, _ := f.Stat()
_ = f.Close()
if fi.Size() < minValidIndexSize { return fmt.Errorf("index truncated: %d bytes", fi.Size()) }
stats, err := block.GatherIndexHealthStats(ctx, bdir)

func blockLooksComplete(bdir string) bool {
    _, err := os.Stat(filepath.Join(bdir, "meta.json"))
    if err != nil { return false }
    fi, err := os.Stat(filepath.Join(bdir, "index"))
    return err == nil && fi.Size() > 0
}

stats, err := block.GatherIndexHealthStats(ctx, bdir)
if err != nil && strings.Contains(err.Error(), "metric label values") {
    return fmt.Errorf("index postings for __name__ unreadable in %s: %w", bdir, err)
}

When it happens

Trigger: GatherIndexHealthStats/VerifyIndex on a block with a truncated index where the __name__ label values postings cannot be decoded, or ctx cancellation mid-read.

Common situations: Incomplete multi-part uploads to object storage, corrupted block downloads (e.g. interrupted prom TSDB block sync), degraded disks.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• lookup label name
• input block index not valid
• get all postings
• label names
• read series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/0c65e46281025447. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:253

		chks []chunks.Meta

		seriesLifeDuration                          = newMinMaxSumInt64()
		seriesLifeDurationWithoutSingleSampleSeries = newMinMaxSumInt64()
		seriesChunks                                = newMinMaxSumInt64()
		chunkDuration                               = newMinMaxSumInt64()
		chunkSize                                   = newMinMaxSumInt64()
		seriesSize                                  = newMinMaxSumInt64()
	)

	lnames, err := r.LabelNames(ctx)
	if err != nil {
		return stats, errors.Wrap(err, "label names")
	}
	stats.LabelNamesCount = int64(len(lnames))

	lvals, err := r.LabelValues(ctx, "__name__", nil)
	if err != nil {
		return stats, errors.Wrap(err, "metric label values")
	}
	stats.MetricLabelValuesCount = int64(len(lvals))

	// As of version two all series entries are 16 byte padded. All references
	// we get have to account for that to get the correct offset.
	offsetMultiplier := 1
	version := r.Version()
	if version >= 2 {
		offsetMultiplier = 16
	}

	// Per series.
	var prevId storage.SeriesRef
	for p.Next() {
		prevLset.CopyFrom(lset)

		id := p.At()
		if prevId != 0 {

View on GitHub (pinned to 35b8b99117)