thanos-io/thanos · error

walk postings

Error message

walk postings

What it means

After walking all postings, GatherIndexHealthStats checks p.Err() and wraps any deferred postings-iteration error as "walk postings". Postings iterators surface corruption asynchronously (checksum/decoding errors mid-iteration), so this catches failures that occur while streaming series rather than at open time.

Solutions

1. Re-download or restore the block from source storage
2. Run VerifyIndex to confirm corruption location
3. Extend context timeouts for large verification sweeps
4. Remove the unreadable block

Example fix

// before
stats, err := block.GatherIndexHealthStats(ctx, ctxBugCtx) // walk postings: context canceled
// after
cctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
stats, err := block.GatherIndexHealthStats(cctx, bdir)

idxPath := filepath.Join(bdir, "index")
fi, err := os.Stat(idxPath)
if err != nil || fi.Size() == 0 { return fmt.Errorf("index missing/empty in %s", bdir) }

stats, err := block.GatherIndexHealthStats(ctx, bdir)
if err != nil && strings.Contains(err.Error(), "walk postings") {
    if errors.Is(ctx.Err(), context.DeadlineExceeded) { return retryLater }
    return fmt.Errorf("block %s postings corrupt: %w", bdir, err)
}

When it happens

Trigger: The postings iterator hits a decoding or checksum failure partway through walking all series in GatherIndexHealthStats, or the context is cancelled during iteration.

Common situations: Bit-rot in index files, truncated block uploads that pass header checks, cancelled long verification runs over many blocks.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• input block index not valid
• get all postings
• label names
• metric label values
• read series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/56b380ec8e2fdbb9. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:377

			ooo++
		}
		if ooo > 0 {
			stats.OutOfOrderSeries++
			stats.OutOfOrderChunks += ooo
			level.Debug(logger).Log("msg", "found out of order series", "labels", lset)
		}

		seriesChunks.Add(int64(len(chks)))
		seriesLifeDuration.Add(seriesLifeTimeMs)

		if seriesLifeTimeMs == 0 {
			stats.SingleSampleSeries++
		} else {
			seriesLifeDurationWithoutSingleSampleSeries.Add(seriesLifeTimeMs)
		}
	}
	if err := p.Err(); err != nil {
		return stats, errors.Wrap(err, "walk postings")
	}

	stats.SeriesMaxLifeDuration = time.Duration(seriesLifeDuration.max) * time.Millisecond
	stats.SeriesAvgLifeDuration = time.Duration(seriesLifeDuration.Avg()) * time.Millisecond
	stats.SeriesMinLifeDuration = time.Duration(seriesLifeDuration.min) * time.Millisecond

	stats.SeriesMaxLifeDurationWithoutSingleSampleSeries = time.Duration(seriesLifeDurationWithoutSingleSampleSeries.max) * time.Millisecond
	stats.SeriesAvgLifeDurationWithoutSingleSampleSeries = time.Duration(seriesLifeDurationWithoutSingleSampleSeries.Avg()) * time.Millisecond
	stats.SeriesMinLifeDurationWithoutSingleSampleSeries = time.Duration(seriesLifeDurationWithoutSingleSampleSeries.min) * time.Millisecond

	stats.SeriesMaxChunks = seriesChunks.max
	stats.SeriesAvgChunks = seriesChunks.Avg()
	stats.SeriesMinChunks = seriesChunks.min

	stats.ChunkMaxSize = chunkSize.max
	stats.ChunkAvgSize = chunkSize.Avg()
	stats.ChunkMinSize = chunkSize.min

View on GitHub (pinned to 35b8b99117)