thanos-io/thanos · error

add symbol

Error message

add symbol

What it means

This error wraps a failure from IndexWriter.AddSymbol() while block.Rewrite (pkg/block/index.go:587) copies all symbols from a source index reader to a new index file during block repair/compaction. AddSymbol fails when the underlying index file writer cannot serialize or flush the symbol table — typically disk I/O errors or a corrupted source symbol iterator. The 'add symbol' wrapper tells you the failure happened in the symbol-copying phase, not the postings or series phase.

Solutions

1. Check disk space and filesystem health (df -h, dmesg) on the directory holding the block; free space or fix the disk and re-run Repair.
2. Verify the source block's index file is not truncated/corrupt; if it is, restore the block from backup or another replica instead of rewriting it.
3. Re-run the repair with a freshly opened destination writer; ensure the IndexWriter was not already closed before rewrite.
4. Inspect the wrapped inner error (errors.Wrap preserves cause) to pinpoint the underlying I/O failure.

Example fix

// before: rewriting onto a nearly full disk
err := block.Rewrite(ctx, meta, blockDir, blockDir, indexr, chunkr, nil)
// after: pre-check free space and source index readability before rewriting
if err := checkDiskSpace(blockDir, requiredBytes); err != nil { return err }
if err := validateIndexFile(filepath.Join(blockDirSrc, "index")); err != nil { return err }
return block.Rewrite(ctx, meta, blockDirSrc, blockDirDst, indexr, chunkr, nil)

if freeDiskSpace(blockDir) < requiredBytes { return fmt.Errorf("insufficient disk space for block rewrite") }
if _, err := os.Stat(filepath.Join(blockDir, "index")); err != nil { return err }

func isAddSymbolError(err error) bool { return err != nil && strings.Contains(err.Error(), "add symbol") }

if err := block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil); err != nil {
    if strings.Contains(err.Error(), "add symbol") {
        // check disk space / writer state, retry with fresh writer
    }
    return err
}

When it happens

Trigger: block.Rewrite() iterating indexr.Symbols() and calling indexw.AddSymbol(symbols.At()) when the destination index file cannot be written (disk full, I/O error, closed writer) or the symbol table serialization fails.

Common situations: Running promtool tsdb repair or block compaction on a disk that is full or failing; corrupted block directories after a crash; repair tooling copying blocks across filesystems with permission or space problems.

Related errors

• next symbol
• index contains postings with out of order labels
• found chunks non-completely outside the block time range…
• postings
• series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/aeb90b6b9f40ed2b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:587

type seriesRepair struct {
	lset labels.Labels
	chks []chunks.Meta
}

// rewrite writes all data from the readers back into the writers while cleaning
// up mis-ordered and duplicated chunks.
func rewrite(
	ctx context.Context,
	logger log.Logger,
	indexr tsdb.IndexReader, chunkr tsdb.ChunkReader,
	indexw tsdb.IndexWriter, chunkw tsdb.ChunkWriter,
	meta *metadata.Meta,
	ignoreChkFns []ignoreFnType,
) error {
	symbols := indexr.Symbols()
	for symbols.Next() {
		if err := indexw.AddSymbol(symbols.At()); err != nil {
			return errors.Wrap(err, "add symbol")
		}
	}
	if symbols.Err() != nil {
		return errors.Wrap(symbols.Err(), "next symbol")
	}

	key, value := index.AllPostingsKey()
	all, err := indexr.Postings(ctx, key, value)
	if err != nil {
		return errors.Wrap(err, "postings")
	}
	all = indexr.SortedPostings(all)

	// We fully rebuild the postings list index from merged series.
	var (
		postings = index.NewMemPostings()
		values   = map[string]stringset{}
		i        = storage.SeriesRef(0)

View on GitHub (pinned to 35b8b99117)