{"record":{"id":"c83974a866e95cf6","repo":"thanos-io/thanos","slug":"q-exceeding-max-size-of-64gib","errorCode":null,"errorMessage":"%q exceeding max size of 64GiB","messagePattern":"%q exceeding max size of 64GiB","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/indexheader/binary_reader.go","lineNumber":385,"sourceCode":"}\n\nfunc (mw *MemoryWriter) Pos() uint64 {\n\treturn mw.pos\n}\n\nfunc (mw *MemoryWriter) Write(bufs ...[]byte) error {\n\tfor _, b := range bufs {\n\t\tn, err := mw.buf.Write(b)\n\t\tmw.pos += uint64(n)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\t// For now the index file must not grow beyond 64GiB. Some of the fixed-sized\n\t\t// offset references in v1 are only 4 bytes large.\n\t\t// Once we move to compressed/varint representations in those areas, this limitation\n\t\t// can be lifted.\n\t\tif mw.pos > 16*math.MaxUint32 {\n\t\t\treturn errors.Errorf(\"%q exceeding max size of 64GiB\", mw.id)\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (mw *MemoryWriter) Buffer() []byte {\n\treturn mw.buf.Bytes()\n}\n\nfunc (mw *MemoryWriter) Flush() error {\n\treturn nil\n}\n\nfunc (mw *MemoryWriter) Sync() error {\n\treturn nil\n}\n\nfunc (mw *MemoryWriter) Close() error {","sourceCodeStart":367,"sourceCodeEnd":403,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/indexheader/binary_reader.go#L367-L403","documentation":"MemoryWriter.Write enforces the TSDB index format v1 hard limit: the index (and thus index-header buffer being written) must stay under 16 * math.MaxUint32 bytes = 64GiB, because v1 fixed-size offset references are only 4 bytes. This error fires when the in-memory writer's position exceeds that limit while writing the binary index-header for a block.","triggerScenarios":"Writing an index-header whose serialized size pushes MemoryWriter.pos past 64GiB — i.e., a block index with an enormous number of series/postings (corrupt or pathologically huge index).","commonSituations":"A block produced by a misbehaving/very old TSDB version with an oversized index; a corrupted index file whose sizes read back absurdly large; ingesting hundreds of millions of series into one block.","solutions":["Verify the block's index size (meta.json / index file) — if it exceeds 64GiB the block is unusable with v1 format; re-compact/split it into smaller blocks.","If the index is unexpectedly huge, treat the block as corrupt: remove it from the bucket and re-upload/re-compact from healthy data.","Upgrade TSDB/Thanos to a version with compressed/varint offset representations if available.","Reduce per-block series count via compaction or sharding so indexes stay well under the limit."],"exampleFix":"null","handlingStrategy":"validation","validationCode":"fi, err := os.Stat(filepath.Join(blockDir, \"index\"))\nif err == nil && fi.Size() > 16*uint64(math.MaxUint32) {\n    return fmt.Errorf(\"block %s index %d bytes exceeds 64GiB limit\", blockID, fi.Size())\n}","typeGuard":"null","tryCatchPattern":"_, err := NewBinaryReader(ctx, logger, bkt, dir, id, sampling, metrics)\nif err != nil && strings.Contains(err.Error(), \"exceeding max size of 64GiB\") {\n    // mark block as unusable / quarantine it, do not retry\n    markBlockCorrupt(id)\n    return err\n}","preventionTips":["Keep compaction block sizes so indexes stay far below 64GiB.","Validate block index size before uploading to the bucket.","Re-compact oversized blocks into smaller time ranges.","Upgrade TSDB to versions with varint offset representations."],"tags":["limit","tsdb","index-header","size-limit"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}