thanos-io/thanos · error

iterate populated chunk series set

Error message

iterate populated chunk series set

What it means

After the full iteration loop, write() checks populatedSet.Err() and wraps it with "iterate populated chunk series set". This is a terminal aggregation error: the populated (materialized) chunk series set hit an error at some point during Next() (e.g. the earlier "get series %d" failure) that propagated to set-level state.

Solutions

1. Unwrap the cause to find the original per-series failure (often the same index-read error logged as 'get series %d').
2. Validate source blocks before compaction (thanos tools bucket verify) to catch corrupt indexes early.
3. Guarantee a single compactor owns the bucket/prefix so blocks aren't deleted mid-run.
4. Retry the compaction after transient storage errors; re-open block readers to clear cached error state.

if err := bucketVerify(ctx, srcBlockIDs); err != nil {
    return fmt.Errorf("pre-compaction block verification failed: %w", err)
}

err := retry.Do(func() error {
    return WriteSeries(ctx, buildPopulatedSet(), symbols)
}, retry.OnRetry(func() { reopenBlockReaders() }), retry.RetryIf(isTransient), retry.Attempts(3))

When it happens

Trigger: WriteSeries finishes (or aborts) iteration over the populated set and populatedSet.Err() is non-nil — typically reflecting an index-read error stored by Next() during series expansion.

Common situations: Any mid-iteration index failure (corrupt index, missing block, IO error) surfaces here if not caught earlier; concurrent block deletion during compaction; object storage timeouts.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• compact series from
• create compactor
• found chunks outside the block time range introduced by…
• compact blocks
• get series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cadb74aa207fcf67. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compactv2/chunk_series_set.go:235

		}

		if err := sWriter.WriteChunks(chks...); err != nil {
			return errors.Wrap(err, "write chunks")
		}
		if err := sWriter.AddSeries(ref, s.Labels(), chks...); err != nil {
			return errors.Wrap(err, "add series")
		}
		for _, chk := range chks {
			// ChunkPool is used by tsdb.OpenBlock BlockReader.
			if err := w.chunkPool.Put(chk.Chunk); err != nil {
				return errors.Wrap(err, "put chunk")
			}
		}
		ref++
		p.SeriesProcessed()
	}
	if populatedSet.Err() != nil {
		return errors.Wrap(populatedSet.Err(), "iterate populated chunk series set")
	}

	return nil
}

View on GitHub (pinned to 35b8b99117)