What causes "open tombstone reader: %w" in thanos-io/thanos?

Read the wrapped %w error for the specific file/IO cause

open tombstone reader

Error message

open tombstone reader: %w

What it means

newBlockBaseQuerier opens the block's tombstone reader via BlockReader.Tombstones() and wraps failure as 'open tombstone reader: %w', closing the previously opened index and chunk readers. A nil tombstone reader is tolerated (replaced by in-memory tombstones), so only an actual error surfaces here.

Solutions

Read the wrapped %w error for the specific file/IO cause
Verify block directory integrity and re-sync the block from object storage if corrupted
Avoid racing queries against blocks being deleted (check compaction/deletion timing)
Check filesystem permissions and health on the block storage path

Defensive patterns

Strategy: retry

Validate before calling

if _, err := os.Stat(filepath.Join(blockDir, "tombstones")); err != nil && !os.IsNotExist(err) { return err }

Type guard

func isTombOpenErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "open tombstone reader")
}

Try / catch

q, err := newBlockBaseQuerier(block, mint, maxt)
if err != nil && strings.Contains(err.Error(), "open tombstone reader") {
    // treat as corrupt block: drop local copy and retry from remote
}

Prevention

Do not delete or rewrite tombstones files while queries are open
After an unclean shutdown, run tsdb verification/repair before serving queries
Ensure consistent permissions on block directories for the process user
Serialize deletions with queries through the block metadata/lease mechanism

When it happens

Trigger: NewCachedBlockChunkQuerier -> newBlockBaseQuerier; b.Tombstones() fails because the tombstones file is corrupt, unreadable, or the block directory changed underneath the reader.

Common situations: Concurrent deletion/compaction of the block while querying, corrupted tombstones file after a crash, permission or I/O problems on the block directory.

Understand the failure class

Background: "open() failed", "failed to open file", "cannot create file" — what a file open error means and how to fix it — this error's family across 42 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/280792de581d69d5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/receive/expandedpostingscache/tsdb.go:54

	mint, maxt int64
}

func newBlockBaseQuerier(b prom_tsdb.BlockReader, mint, maxt int64) (*blockBaseQuerier, error) {
	indexr, err := b.Index()
	if err != nil {
		return nil, fmt.Errorf("open index reader: %w", err)
	}
	chunkr, err := b.Chunks()
	if err != nil {
		indexr.Close()
		return nil, fmt.Errorf("open chunk reader: %w", err)
	}
	tombsr, err := b.Tombstones()
	if err != nil {
		indexr.Close()
		chunkr.Close()
		return nil, fmt.Errorf("open tombstone reader: %w", err)
	}

	if tombsr == nil {
		tombsr = tombstones.NewMemTombstones()
	}
	return &blockBaseQuerier{
		blockID:    b.Meta().ULID,
		mint:       mint,
		maxt:       maxt,
		index:      indexr,
		chunks:     chunkr,
		tombstones: tombsr,
	}, nil
}

func (q *blockBaseQuerier) LabelValues(ctx context.Context, name string, hints *storage.LabelHints, matchers ...*labels.Matcher) ([]string, annotations.Annotations, error) {
	res, err := q.index.SortedLabelValues(ctx, name, hints, matchers...)
	return res, nil, err

View on GitHub (pinned to 35b8b99117)