thanos-io/thanos · error

invalid remaining size, even after refetch, remaining

Error message

invalid remaining size, even after refetch, remaining: %d, expected %d

What it means

During series decoding, if the uvarint length prefix says the series record extends beyond the fetched byte range (len(c) < n+int(l)), the reader first tries one refetch with a larger estimated size. If it still does not fit on the first id with refetch allowed, this error is returned: even after enlarging the read, the series does not fit the fetched range, meaning the range/size estimate or the index bytes are wrong.

Solutions

1. Increase store-gateway's --store.index-header-postings-offsets-max-size / max-series-size estimates so refetch fetches enough bytes
2. Verify the block and delete/re-upload it if its index is corrupt
3. Ensure postings (refs) and series are read from the same block version — avoid querying a block that was replaced
4. Retry with a fresh reader so postings offsets and index header are re-fetched consistently

if estimatedMaxSeriesSize == 0 {
	estimatedMaxSeriesSize = defaultMaxSeriesSize // ensure refetch has room
}
if uint64(n)+uint64(l) > maxAllowedSeriesSize {
	// bogus length: corrupt index, abort rather than looping
}

func plausibleSeriesLen(n, l int, max int) bool { return n > 0 && l >= 0 && n+l <= max }

if len(c) < n+int(l) {
	if i == 0 && refetch {
		logger.Warn("series does not fit even after refetch; invalidating block cache", "id", id)
		invalidateBlockCache(blockMeta.ULID)
		return retryLoadSeries(ctx, ids) // one clean retry with fresh estimates
	}
}

When it happens

Trigger: First series id in the batch requires more bytes than fetched and r.block.readIndexRange was already retried with an expanded end: index layout changed (block replaced), estimatedMaxSeriesSize far too small, or corrupted index producing a bogus huge length.

Common situations: Very large single series entries exceeding maxSeriesSize estimation; block replaced in the bucket with a different index while old postings refs are still used; corrupt index giving nonsensical lengths; misconfigured max-series-size-bytes in store-gateway.

Related errors

• reading series length failed
• read series
• series
• iterate series
• repaired block is invalid

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/13ccf289fa4f8d6c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:3415

	b, err := r.block.readIndexRange(ctx, int64(start), int64(end-start), r.logger)
	if err != nil {
		return errors.Wrap(err, "read series range")
	}

	stats.seriesFetchCount++
	stats.SeriesFetchDurationSum += time.Since(begin)
	stats.add(SeriesFetched, len(ids), int(end-start))

	for i, id := range ids {
		c := b[uint64(id)-start:]

		l, n := binary.Uvarint(c)
		if n < 1 {
			return errors.New("reading series length failed")
		}
		if len(c) < n+int(l) {
			if i == 0 && refetch {
				return errors.Errorf("invalid remaining size, even after refetch, remaining: %d, expected %d", len(c), n+int(l))
			}

			// Inefficient, but should be rare.
			r.block.metrics.seriesRefetches.WithLabelValues(tenant).Inc()
			level.Warn(r.logger).Log("msg", "series size exceeded expected size; refetching", "id", id, "series length", n+int(l), "maxSeriesSize", r.block.estimatedMaxSeriesSize)

			// Fetch plus to get the size of next one if exists.
			return r.loadSeries(ctx, ids[i:], true, uint64(id), uint64(id)+uint64(n+int(l)+1), bytesLimiter, tenant)
		}
		c = c[n : n+int(l)]
		r.loadedSeriesMtx.Lock()
		r.loadedSeries[id] = c
		r.loadedSeriesMtx.Unlock()

		r.block.indexCache.StoreSeries(r.block.meta.ULID, id, c, tenant)
	}
	return nil
}

View on GitHub (pinned to 35b8b99117)