thanos-io/thanos · error

populate chunk

Error message

populate chunk

What it means

Raised in bucketChunkReader.load when decoding the chunk header yields chunkDataLen but converting/decoding the chunk (b.Toolbox decode of chunk bytes via chunkenc) fails — the chunk bytes fetched are corrupt or the encoding byte is invalid, so the chunk cannot be materialized.

Solutions

1. Check the block's chunks files for corruption (verify checksums)
2. Re-upload or restore the affected block from a backup
3. Identify the series/chunk via the wrapped error and query a different replica if available

When it happens

Trigger: Thrown at pkg/store/bucket.go:3800 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4665b7ded217ac2d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:3800

		readOffset += n
		// Unexpected EOF for last chunk could be a valid case. Any other errors are definitely real.
		if err != nil && !errors.Is(err, io.ErrUnexpectedEOF) && i != len(pIdxs)-1 {
			return errors.Wrapf(err, "read range for seq %d offset %x", seq, pIdx.offset)
		}

		chunkDataLen, n := binary.Uvarint(cb)
		if n < 1 {
			return errors.New("reading chunk length failed")
		}

		// Chunk length is n (number of bytes used to encode chunk data), 1 for chunk encoding and chunkDataLen for actual chunk data.
		// There is also crc32 after the chunk, but we ignore that.
		chunkLen = n + 1 + int(chunkDataLen)
		if chunkLen <= len(cb) {
			c := rawChunk(cb[n:chunkLen])
			err = populateChunk(&(res[pIdx.seriesEntry].chks[pIdx.chunk]), &c, aggrs, r.save, calculateChunkChecksum)
			if err != nil {
				return errors.Wrap(err, "populate chunk")
			}
			stats.add(ChunksTouched, 1, int(chunkDataLen))
			continue
		}

		r.block.metrics.chunkRefetches.WithLabelValues(tenant).Inc()
		// If we didn't fetch enough data for the chunk, fetch more.
		fetchBegin = time.Now()
		// Read entire chunk into new buffer.
		// TODO: readChunkRange call could be avoided for any chunk but last in this particular part.
		if err := bytesLimiter.ReserveWithType(uint64(chunkLen), ChunksTouched); err != nil {
			return httpgrpc.Errorf(int(codes.ResourceExhausted), "exceeded bytes limit while fetching chunks: %s", err)
		}

		nb, err := r.block.readChunkRange(ctx, seq, int64(pIdx.offset), int64(chunkLen), []byteRange{{offset: 0, length: chunkLen}}, r.logger)
		if err != nil {
			return errors.Wrapf(err, "preloaded chunk too small, expecting %d, and failed to fetch full chunk", chunkLen)
		}

View on GitHub (pinned to 35b8b99117)