{"record":{"id":"13ccf289fa4f8d6c","repo":"thanos-io/thanos","slug":"invalid-remaining-size-even-after-refetch-remain","errorCode":null,"errorMessage":"invalid remaining size, even after refetch, remaining: %d, expected %d","messagePattern":"invalid remaining size, even after refetch, remaining: (.+?), expected (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/bucket.go","lineNumber":3415,"sourceCode":"\tb, err := r.block.readIndexRange(ctx, int64(start), int64(end-start), r.logger)\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"read series range\")\n\t}\n\n\tstats.seriesFetchCount++\n\tstats.SeriesFetchDurationSum += time.Since(begin)\n\tstats.add(SeriesFetched, len(ids), int(end-start))\n\n\tfor i, id := range ids {\n\t\tc := b[uint64(id)-start:]\n\n\t\tl, n := binary.Uvarint(c)\n\t\tif n < 1 {\n\t\t\treturn errors.New(\"reading series length failed\")\n\t\t}\n\t\tif len(c) < n+int(l) {\n\t\t\tif i == 0 && refetch {\n\t\t\t\treturn errors.Errorf(\"invalid remaining size, even after refetch, remaining: %d, expected %d\", len(c), n+int(l))\n\t\t\t}\n\n\t\t\t// Inefficient, but should be rare.\n\t\t\tr.block.metrics.seriesRefetches.WithLabelValues(tenant).Inc()\n\t\t\tlevel.Warn(r.logger).Log(\"msg\", \"series size exceeded expected size; refetching\", \"id\", id, \"series length\", n+int(l), \"maxSeriesSize\", r.block.estimatedMaxSeriesSize)\n\n\t\t\t// Fetch plus to get the size of next one if exists.\n\t\t\treturn r.loadSeries(ctx, ids[i:], true, uint64(id), uint64(id)+uint64(n+int(l)+1), bytesLimiter, tenant)\n\t\t}\n\t\tc = c[n : n+int(l)]\n\t\tr.loadedSeriesMtx.Lock()\n\t\tr.loadedSeries[id] = c\n\t\tr.loadedSeriesMtx.Unlock()\n\n\t\tr.block.indexCache.StoreSeries(r.block.meta.ULID, id, c, tenant)\n\t}\n\treturn nil\n}","sourceCodeStart":3397,"sourceCodeEnd":3433,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/bucket.go#L3397-L3433","documentation":"During series decoding, if the uvarint length prefix says the series record extends beyond the fetched byte range (len(c) < n+int(l)), the reader first tries one refetch with a larger estimated size. If it still does not fit on the first id with refetch allowed, this error is returned: even after enlarging the read, the series does not fit the fetched range, meaning the range/size estimate or the index bytes are wrong.","triggerScenarios":"First series id in the batch requires more bytes than fetched and r.block.readIndexRange was already retried with an expanded end: index layout changed (block replaced), estimatedMaxSeriesSize far too small, or corrupted index producing a bogus huge length.","commonSituations":"Very large single series entries exceeding maxSeriesSize estimation; block replaced in the bucket with a different index while old postings refs are still used; corrupt index giving nonsensical lengths; misconfigured max-series-size-bytes in store-gateway.","solutions":["Increase store-gateway's --store.index-header-postings-offsets-max-size / max-series-size estimates so refetch fetches enough bytes","Verify the block and delete/re-upload it if its index is corrupt","Ensure postings (refs) and series are read from the same block version — avoid querying a block that was replaced","Retry with a fresh reader so postings offsets and index header are re-fetched consistently"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if estimatedMaxSeriesSize == 0 {\n\testimatedMaxSeriesSize = defaultMaxSeriesSize // ensure refetch has room\n}\nif uint64(n)+uint64(l) > maxAllowedSeriesSize {\n\t// bogus length: corrupt index, abort rather than looping\n}","typeGuard":"func plausibleSeriesLen(n, l int, max int) bool { return n > 0 && l >= 0 && n+l <= max }","tryCatchPattern":"if len(c) < n+int(l) {\n\tif i == 0 && refetch {\n\t\tlogger.Warn(\"series does not fit even after refetch; invalidating block cache\", \"id\", id)\n\t\tinvalidateBlockCache(blockMeta.ULID)\n\t\treturn retryLoadSeries(ctx, ids) // one clean retry with fresh estimates\n\t}\n}","preventionTips":["Keep max-series-size-bytes estimates aligned with actual data","Never reuse ULIDs when rebuilding blocks","Invalidate per-block caches on block replacement","Monitor seriesRefetches metric for systematic underestimation"],"tags":["thanos","series","index","corruption","bounds"],"backgroundTag":"series-refetch-size-mismatch","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}