thanos-io/thanos · error

postings

Error message

postings

What it means

This error wraps a failure from IndexReader.Postings(ctx, key, value) in block.Rewrite (pkg/block/index.go:597), where the code fetches the all-postings list using index.AllPostingsKey() to rebuild the postings index. It indicates the source index file's postings section cannot be read or decoded — a corrupt/truncated postings offset table — rather than a symbol or series problem.

Solutions

1. Restore the block from backup/replica or delete the corrupt block and let TSDB re-ingest or re-compact from upstream data.
2. Use promtool tsdb dump/analyze to confirm which index sections are readable and gauge data loss scope.
3. Check the block's index file size and checksums if the block was copied between nodes; re-copy if truncated.
4. Verify the reading Prometheus/promtool version is compatible with the block's index format version.

Example fix

// before: rewrite proceeds and fails on unreadable postings
err := block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil)
// after: pre-validate that all-postings are readable before rewrite
_, err := indexr.Postings(ctx, index.AllPostingsKey())
if err != nil { return fmt.Errorf("source index unreadable, restore block: %w", err) }
return block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil)

all, err := indexr.Postings(ctx, index.AllPostingsKey())
if err != nil { return fmt.Errorf("source index postings unreadable; restore block: %w", err) }
defer all.Close()

func isPostingsError(err error) bool { return err != nil && strings.Contains(err.Error(), "postings") }

if err := block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil); err != nil {
    if strings.Contains(err.Error(), "postings") {
        // postings section corrupt: quarantine block, do not retry rewrite
    }
    return err
}

When it happens

Trigger: indexr.Postings(ctx, AllPostingsName, AllPostingsListValue) failing while reading the postings offset table for the special all-postings key during rewrite/Repair.

Common situations: Blocks damaged by crash during compaction; disk-level corruption; manually copied blocks missing parts of the index file; promtool repair on blocks produced by an older Prometheus index format.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• next symbol
• series
• iterate series
• read postings table
• get postings offset entry

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9d708f66230606d4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:597

	indexr tsdb.IndexReader, chunkr tsdb.ChunkReader,
	indexw tsdb.IndexWriter, chunkw tsdb.ChunkWriter,
	meta *metadata.Meta,
	ignoreChkFns []ignoreFnType,
) error {
	symbols := indexr.Symbols()
	for symbols.Next() {
		if err := indexw.AddSymbol(symbols.At()); err != nil {
			return errors.Wrap(err, "add symbol")
		}
	}
	if symbols.Err() != nil {
		return errors.Wrap(symbols.Err(), "next symbol")
	}

	key, value := index.AllPostingsKey()
	all, err := indexr.Postings(ctx, key, value)
	if err != nil {
		return errors.Wrap(err, "postings")
	}
	all = indexr.SortedPostings(all)

	// We fully rebuild the postings list index from merged series.
	var (
		postings = index.NewMemPostings()
		values   = map[string]stringset{}
		i        = storage.SeriesRef(0)
		series   = []seriesRepair{}
	)

	var builder labels.ScratchBuilder
	var chks []chunks.Meta
	for all.Next() {
		id := all.At()

		if err := indexr.Series(id, &builder, &chks); err != nil {
			return errors.Wrap(err, "series")

View on GitHub (pinned to 35b8b99117)