thanos-io/thanos · error
add symbol
Error message
add symbol
What it means
During compaction write-out, AddSymbol on the destination index writer failed for one of the series' symbols. The index writer rejects or fails to persist symbols (deduplicated symbol table writes), aborting the write phase of compaction.
Solutions
- Check free disk space on the compactor's scratch/data volume — full disks are the most common cause of index write failures.
- Inspect the wrapped cause; if it indicates an internal invariant (double add), verify the symbols iterator yields unique symbols.
- Retry compaction with a fresh destination index writer; a failed partial index must be discarded, not appended to.
- Verify block versions compatibility between source index and destination writer.
Defensive patterns
Strategy: validation
Validate before calling
if freeDiskBytes(dataDir) < minCompactionScratchBytes {
return fmt.Errorf("insufficient scratch space: %d bytes free", freeDiskBytes(dataDir))
} Try / catch
if err := compact(ctx); err != nil {
if strings.Contains(errors.Cause(err).Error(), "no space left") {
cleanupPartialOutput(); return retryWithMoreSpace(err)
}
return err
} Prevention
- Monitor scratch/data volume free space and alert before it fills
- Discard and rebuild partial output indexes — never append to a failed writer
- Ensure symbol iterators yield unique symbols if using custom index code
- Keep compactor temp dirs on healthy local disks, not network mounts
When it happens
Trigger: write() iterating symbols.Next() and calling sWriter.AddSymbol(symbols.At()) when the destination index writer errors: writing to a closed/full index file, symbol already added with a mismatched state (internal writer invariant), or underlying IO failure.
Common situations: Disk full on the compactor's scratch volume, adding the same symbol twice due to a duplicate symbol iterator (custom/patched index), index writer used after the block was already finalized.
Understand the failure class
Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.
Related errors
AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07).
Data as JSON: /api/errors/42aaded66955bcf3.
Report an issue: GitHub.
Appendix: source
Thrown at pkg/compactv2/chunk_series_set.go:184
l.populated.Reset(stream)
}
func (l *lazyPopulatableChunk) Compact() {
if l.populated == nil {
l.populate()
}
l.populated.Compact()
}
func (w *Compactor) write(ctx context.Context, symbols index.StringIter, populatedSet storage.ChunkSeriesSet, sWriter block.SeriesWriter, p ProgressLogger) error {
var (
chks []chunks.Meta
ref storage.SeriesRef
)
for symbols.Next() {
if err := sWriter.AddSymbol(symbols.At()); err != nil {
return errors.Wrap(err, "add symbol")
}
}
if err := symbols.Err(); err != nil {
return errors.Wrap(err, "symbols")
}
// Iterate over all sorted chunk series.
for populatedSet.Next() {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
s := populatedSet.At()
chksIter := s.Iterator(nil)
chks = chks[:0]
for chksIter.Next() {View on GitHub (pinned to 35b8b99117)