thanos-io/thanos · error

cannot populate chunk

Error message

cannot populate chunk %d

What it means

lazyPopulatableChunk.populate loads the actual chunk bytes via ChunkOrIterable for a chunk meta; on failure the chunk is replaced with an errChunk holding the wrapped error. The error surfaces later as a chunk iterator error when the data is read, referencing the chunk's Ref. It's a deferred failure: population happens lazily during compaction materialization.

Solutions

1. Inspect the inner error from the errChunk iterator: treat corruption by deleting/repairing the offending source block.
2. Verify chunk file integrity (chunk checksums) and re-sync the block from a healthy replica.
3. Ensure the compactor reads blocks that are fully uploaded (check upload completion markers; avoid compacting partial blocks).
4. Retry compaction after storage issues resolve; if persistent, exclude the bad block via ignore-blocks or re-upload it.

Example fix

// the library defers the error into an errChunk iterator:
l.m.Chunk = errChunk{err: errChunkIterator{err: errors.Wrapf(err, "cannot populate chunk %d", l.m.Ref)}}
// caller should surface it instead of silently dropping samples:
it := chk.Chunk.Iterator(nil)
for it.Next() != chunkenc.ValNone {
    if it.Err() != nil {
        return errors.Wrapf(it.Err(), "reading chunk %d", chk.Ref)
    }
}

// check chunk data availability before compacting the block
if err := validateChunkFiles(blockDir); err != nil {
    return fmt.Errorf("block %s has corrupt chunk data: %w", blockDir, err)
}

func chunkIsReadable(m chunks.Meta, cr *blockChunkReader) error {
    _, _, err := cr.ChunkOrIterable(m)
    return err
}

defer func() {
    if rec := recover(); rec != nil { _ = rec }
}()
it := chk.Chunk.Iterator(nil)
for it.Next() != chunkenc.ValNone {
    if err := it.Err(); err != nil {
        return errors.Wrapf(err, "chunk %d unreadable", chk.Ref)
    }
}

When it happens

Trigger: Calling populate (transitively via Bytes/Encoding/Iterator/NumSamples) on a chunk whose ChunkOrIterable call fails — typically reading the chunk from the source block's chunk files fails: bad offset in chunk meta, truncated/corrupt chunk segment file, IO error.

Common situations: Truncated chunks files from interrupted uploads to object storage, disk corruption, chunk refs pointing at a different (incompatible) block after manual block manipulation, version-mismatched block format.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• get series
• chunk iter
• write chunks
• open block
• open block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/64e4c4c29a3924d6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compactv2/chunk_series_set.go:120

func (e errChunkIterator) Err() error               { return e.err }

type errChunk struct{ err errChunkIterator }

func (e errChunk) Bytes() []byte                                { return nil }
func (e errChunk) Encoding() chunkenc.Encoding                  { return chunkenc.EncXOR }
func (e errChunk) Appender() (chunkenc.Appender, error)         { return nil, e.err.err }
func (e errChunk) Iterator(chunkenc.Iterator) chunkenc.Iterator { return e.err }
func (e errChunk) NumSamples() int                              { return 0 }
func (e errChunk) Compact()                                     {}
func (e errChunk) Reset(stream []byte)                          {}

func (l *lazyPopulatableChunk) populate() {
	// TODO(bwplotka): In most cases we don't need to parse anything, just copy. Extend reader/writer for this.
	var err error
	// Ignore iterable as it should be nil.
	l.populated, _, err = l.cr.ChunkOrIterable(*l.m)
	if err != nil {
		l.m.Chunk = errChunk{err: errChunkIterator{err: errors.Wrapf(err, "cannot populate chunk %d", l.m.Ref)}}
		return
	}

	l.m.Chunk = l.populated
}

func (l *lazyPopulatableChunk) Bytes() []byte {
	if l.populated == nil {
		l.populate()
	}
	return l.populated.Bytes()
}

func (l *lazyPopulatableChunk) Encoding() chunkenc.Encoding {
	if l.populated == nil {
		l.populate()
	}
	return l.populated.Encoding()

View on GitHub (pinned to 35b8b99117)