thanos-io/thanos · error
exceeding max size of 64GiB
Error message
%q exceeding max size of 64GiB
What it means
MemoryWriter.Write enforces the TSDB index format v1 hard limit: the index (and thus index-header buffer being written) must stay under 16 * math.MaxUint32 bytes = 64GiB, because v1 fixed-size offset references are only 4 bytes. This error fires when the in-memory writer's position exceeds that limit while writing the binary index-header for a block.
Solutions
- Verify the block's index size (meta.json / index file) — if it exceeds 64GiB the block is unusable with v1 format; re-compact/split it into smaller blocks.
- If the index is unexpectedly huge, treat the block as corrupt: remove it from the bucket and re-upload/re-compact from healthy data.
- Upgrade TSDB/Thanos to a version with compressed/varint offset representations if available.
- Reduce per-block series count via compaction or sharding so indexes stay well under the limit.
Example fix
null
Defensive patterns
Strategy: validation
Validate before calling
fi, err := os.Stat(filepath.Join(blockDir, "index"))
if err == nil && fi.Size() > 16*uint64(math.MaxUint32) {
return fmt.Errorf("block %s index %d bytes exceeds 64GiB limit", blockID, fi.Size())
} Type guard
null
Try / catch
_, err := NewBinaryReader(ctx, logger, bkt, dir, id, sampling, metrics)
if err != nil && strings.Contains(err.Error(), "exceeding max size of 64GiB") {
// mark block as unusable / quarantine it, do not retry
markBlockCorrupt(id)
return err
} Prevention
- Keep compaction block sizes so indexes stay far below 64GiB.
- Validate block index size before uploading to the bucket.
- Re-compact oversized blocks into smaller time ranges.
- Upgrade TSDB to versions with varint offset representations.
When it happens
Trigger: Writing an index-header whose serialized size pushes MemoryWriter.pos past 64GiB — i.e., a block index with an enormous number of series/postings (corrupt or pathologically huge index).
Common situations: A block produced by a misbehaving/very old TSDB version with an oversized index; a corrupted index file whose sizes read back absurdly large; ingesting hundreds of millions of series into one block.
Understand the failure class
Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.
Related errors
AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07).
Data as JSON: /api/errors/c83974a866e95cf6.
Report an issue: GitHub.
Appendix: source
Thrown at pkg/block/indexheader/binary_reader.go:385
}
func (mw *MemoryWriter) Pos() uint64 {
return mw.pos
}
func (mw *MemoryWriter) Write(bufs ...[]byte) error {
for _, b := range bufs {
n, err := mw.buf.Write(b)
mw.pos += uint64(n)
if err != nil {
return err
}
// For now the index file must not grow beyond 64GiB. Some of the fixed-sized
// offset references in v1 are only 4 bytes large.
// Once we move to compressed/varint representations in those areas, this limitation
// can be lifted.
if mw.pos > 16*math.MaxUint32 {
return errors.Errorf("%q exceeding max size of 64GiB", mw.id)
}
}
return nil
}
func (mw *MemoryWriter) Buffer() []byte {
return mw.buf.Bytes()
}
func (mw *MemoryWriter) Flush() error {
return nil
}
func (mw *MemoryWriter) Sync() error {
return nil
}
func (mw *MemoryWriter) Close() error {View on GitHub (pinned to 35b8b99117)