thanos-io/thanos · error

get series

Error message

get series %d

What it means

In compactor's chunkSeriesSet.Next, expanding a postings list ref into a series via the block's index reader (ir.Series) failed. The ref is wrapped into s.err and iteration stops. storage.ErrNotFound is explicitly tolerated (stale postings), so this error means any index-read failure other than a missing series.

Solutions

1. Inspect the wrapped error: if it's an index checksum/decode failure, delete the corrupt block from storage (Thanos will repair by re-uploading or remove from view).
2. Ensure no concurrent deletion of the source blocks while compaction runs (single compactor per bucket prefix).
3. Retry the compaction; transient IO errors may clear, but re-open the block reader first (index readers may cache failures).
4. If it happens reproducibly after partial uploads, run a block repair/sync to restore consistent block data.

Example fix

// handled inside the library already:
if errors.Cause(err) == storage.ErrNotFound {
    continue // stale postings tolerated
}
s.err = errors.Wrapf(err, "get series %d", s.all.At())
// caller-side: check and propagate
if set.Err() != nil {
    return errors.Wrap(set.Err(), "iterating series during compaction")
}

if err := verifyBlockIndex(blockDir); err != nil {
    // exclude corrupt block from compaction set
    skipBlock(id)
}

func isStalePostings(err error) bool {
    return errors.Cause(err) == storage.ErrNotFound
}

if err := runCompaction(ctx); err != nil {
    var idxErr *indexErr
    if errors.As(err, &idxErr) {
        repairOrDeleteCorruptBlock(idxErr.BlockID)
    }
    return err
}

When it happens

Trigger: Calling Next() on the sorted chunk series set during compaction when ir.Series(ref, labels, chunks) returns a non-NotFound error: index decode failure, corrupt index file, IO error reading the block's index, or a race where the block is removed mid-compaction.

Common situations: Corrupted TSDB index after a crashed write or bad disk, block files deleted while compaction reads them, checksum failures on block index files, reading blocks from partially-uploaded object storage.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• next symbol
• postings
• series
• iterate series
• cannot populate chunk

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fba9c97ba2399b92. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compactv2/chunk_series_set.go:45

	bufChks []chunks.Meta
	bufLbls labels.ScratchBuilder

	curr *storage.ChunkSeriesEntry
	err  error
}

func newLazyPopulateChunkSeriesSet(sReader seriesReader, all index.Postings) *lazyPopulateChunkSeriesSet {
	return &lazyPopulateChunkSeriesSet{sReader: sReader, all: all}
}

func (s *lazyPopulateChunkSeriesSet) Next() bool {
	for s.all.Next() {
		if err := s.sReader.ir.Series(s.all.At(), &s.bufLbls, &s.bufChks); err != nil {
			// Postings may be stale. Skip if no underlying series exists.
			if errors.Cause(err) == storage.ErrNotFound {
				continue
			}
			s.err = errors.Wrapf(err, "get series %d", s.all.At())
			return false
		}

		if len(s.bufChks) == 0 {
			continue
		}

		for i := range s.bufChks {
			s.bufChks[i].Chunk = &lazyPopulatableChunk{cr: s.sReader.cr, m: &s.bufChks[i]}
		}
		s.curr = &storage.ChunkSeriesEntry{
			Lset: s.bufLbls.Labels(),
			ChunkIteratorFn: func(_ chunks.Iterator) chunks.Iterator {
				return storage.NewListChunkSeriesIterator(s.bufChks...)
			},
		}
		return true
	}

View on GitHub (pinned to 35b8b99117)