thanos-io/thanos · error

decode postings

Error message

decode postings

What it means

bucketIndexReader.Postings failed to decode a cached postings blob fetched from the block index cache. The cache stores snappy-compressed diff-varint postings, and decodeCachedPostings snappy-decodes and parses them back into a Postings list. This error means the cached bytes are unreadable/corrupt or were written by an incompatible format version, so the reader cannot serve postings from cache.

Solutions

1. Evict the affected postings cache entries (flush or restart the cache) so postings are re-fetched and re-encoded from the index
2. Verify all Thanos store-gateway instances run the same version so cache encoding is consistent
3. Disable or clear the external postings cache (store.index-cache.posts config) and re-enable after upgrade
4. Check the cache backend for binary-unsafe serialization or size limits that truncate values

Example fix

// before
l, closer, err := r.decodeCachedPostings(b)
if err != nil {
	return nil, closeFns, errors.Wrap(err, "decode postings")
}
// after
l, closer, err := r.decodeCachedPostings(b)
if err != nil {
	level.Warn(r.logger).Log("msg", "failed to decode cached postings, falling back to fetch", "err", err)
	r.block.indexCache.StorePostings(...) // invalidate/overwrite stale entry
	// fall through to remote fetch path instead of failing the query
}

if len(cached) == 0 || !bytes.HasPrefix(cached, snappyStreamedMagic) {
	// treat as cache miss: fall back to fetching postings from object storage
}

func isValidCachedPostings(b []byte) bool { return len(b) > 0 && isSnappyStreamed(b) }

l, closer, err := r.decodeCachedPostings(b)
if err != nil {
	logger.Warn("cached postings undecodable; refetching from index", "err", err)
	l, closer, err = fetchPostingsFromIndex(ctx, key) // fallback path
	if err != nil { return nil, closeFns, err }
}

When it happens

Trigger: r.block.indexCache.FetchPostings returned bytes for a key and r.decodeCachedPostings(b) returned an error: corrupted cache entry in the shared store/index cache, cache entry produced by a different Thanos version or encoding scheme (e.g. plain snappy vs streamed snappy, or future cachedPostingsVersion), or truncated cache payload.

Common situations: Upgrading Thanos while an object-storage or in-memory index cache still holds entries encoded by the old version; a buggy or misconfigured cache backend (e.g. Redis/Memcached with evictions, truncation, or binary-corrupting serialization); manual tampering with cached blobs.

Related errors

• encoding with snappy
• reading postings
• reading series length failed
• iterate series
• repaired block is invalid

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e00175c8b2eac6fd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:3150

		return nil, closeFns, httpgrpc.Errorf(int(codes.ResourceExhausted), "exceeded bytes limit while loading postings from index cache: %s", err)
	}

	// Iterate over all groups and fetch posting from cache.
	// If we have a miss, mark key to be fetched in `ptrs` slice.
	// Overlaps are well handled by partitioner, so we don't need to deduplicate keys.
	for ix, key := range keys {
		if (ix+1)%checkContextEveryNIterations == 0 {
			if err := ctx.Err(); err != nil {
				return nil, closeFns, err
			}
		}
		// Get postings for the given key from cache first.
		if b, ok := fromCache[key]; ok {
			r.stats.add(PostingsTouched, 1, len(b))

			l, closer, err := r.decodeCachedPostings(b)
			if err != nil {
				return nil, closeFns, errors.Wrap(err, "decode postings")
			}
			output[ix] = l
			closeFns = append(closeFns, closer...)
			continue
		}

		// Cache miss; save pointer for actual posting in index stored in object store.
		ptr, err := r.block.indexHeaderReader.PostingsOffset(key.Name, key.Value)
		if err == indexheader.NotFoundRangeErr {
			// This block does not have any posting for given key.
			output[ix] = index.EmptyPostings()
			continue
		}

		if err != nil {
			return nil, closeFns, errors.Wrap(err, "index header PostingsOffset")
		}

View on GitHub (pinned to 35b8b99117)