thanos-io/thanos · error

next symbol

Error message

next symbol

What it means

This error wraps symbols.Err() after iterating the source index's symbol table in block.Rewrite (pkg/block/index.go:591). It means the Symbol iterator itself returned an error while reading the symbol table from the source index file — i.e. the source index is truncated or corrupt at the symbol section, not a write failure. Raised in the same Repair/TestRewrite path as 'add symbol' but signals a read-side problem.

Solutions

1. Treat the source block as corrupt: restore it from backup, another replica, or re-fetch via Thanos/Cortex compactor before attempting rewrite.
2. Run promtool tsdb analyze on the block to confirm the index file is unreadable, then delete/quarantine the block so TSDB can repair metadata.
3. Check for truncated 'index' files (compare size against meta.json expectations) and re-copy the block.
4. If the error persists on valid-looking blocks, check for Prometheus/storage version mismatch producing an index layout the reader cannot decode.

Example fix

// before: blindly repairing a possibly truncated block
if err := block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil); err != nil { return err }
// after: validate index file integrity first
fi, err := os.Stat(filepath.Join(src, "index"))
if err != nil || fi.Size() < minIndexSize { return fmt.Errorf("index truncated (%d bytes), restore block", fi.Size()) }
return block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil)

fi, err := os.Stat(filepath.Join(blockDir, "index"))
if err != nil { return err }
if fi.Size() == 0 { return fmt.Errorf("index file truncated/empty: %s", blockDir) }

func isSymbolIterError(err error) bool { return err != nil && strings.Contains(err.Error(), "next symbol") }

if err := block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil); err != nil {
    if strings.Contains(err.Error(), "next symbol") {
        // source index corrupt: restore from replica/backup instead of retrying
    }
    return err
}

When it happens

Trigger: After symbols.Next() returns false, symbols.Err() is non-nil because reading the symbol table slice from the source index file failed (bad offset, truncated file, CRC/decode failure in the symbols section).

Common situations: Repairing blocks from a node that crashed mid-write; blocks copied with rsync/scp interrupted leaving a truncated index file; filesystem corruption after power loss; promtool tsdb repair on damaged historical blocks.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• add symbol
• postings
• series
• chunk read
• iterate series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e5a93010ce9c0d2a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:591

// rewrite writes all data from the readers back into the writers while cleaning
// up mis-ordered and duplicated chunks.
func rewrite(
	ctx context.Context,
	logger log.Logger,
	indexr tsdb.IndexReader, chunkr tsdb.ChunkReader,
	indexw tsdb.IndexWriter, chunkw tsdb.ChunkWriter,
	meta *metadata.Meta,
	ignoreChkFns []ignoreFnType,
) error {
	symbols := indexr.Symbols()
	for symbols.Next() {
		if err := indexw.AddSymbol(symbols.At()); err != nil {
			return errors.Wrap(err, "add symbol")
		}
	}
	if symbols.Err() != nil {
		return errors.Wrap(symbols.Err(), "next symbol")
	}

	key, value := index.AllPostingsKey()
	all, err := indexr.Postings(ctx, key, value)
	if err != nil {
		return errors.Wrap(err, "postings")
	}
	all = indexr.SortedPostings(all)

	// We fully rebuild the postings list index from merged series.
	var (
		postings = index.NewMemPostings()
		values   = map[string]stringset{}
		i        = storage.SeriesRef(0)
		series   = []seriesRepair{}
	)

	var builder labels.ScratchBuilder

View on GitHub (pinned to 35b8b99117)