thanos-io/thanos · error

unrecognize postings format

Error message

unrecognize postings format

What it means

decodePostings inspects the input byte slice for known codec magic headers (diff+varint+snappy, or its streamed variant) and returns this error when none match. It means the cached postings bytes were not produced by any supported encoder — the format is unknown or the data is corrupt.

Solutions

1. Clear the affected postings cache (index-cache) so entries are re-encoded with the current codec
2. Ensure the Prometheus version reading the cache is >= the version that wrote it
3. Verify storage integrity; if index data is corrupt, restore from backup or re-ingest
4. Check for external tools writing directly to the cache files

Example fix

// before: trusting arbitrary cache bytes
df, _ := decodeCachedPostings(b)
// after: validate header before decoding
if !isDiffVarintSnappyEncodedPostings(b) && !isDiffVarintSnappyStreamedEncodedPostings(b) {
    return fmt.Errorf("unrecognized postings format, purge cache")
}

func decodablePostings(b []byte) bool {
    return isDiffVarintSnappyEncodedPostings(b) || isDiffVarintSnappyStreamedEncodedPostings(b)
}

func hasKnownPostingsHeader(b []byte) bool {
    return bytes.HasPrefix(b, codecHeaderStreamedSnappy) || isDiffVarintSnappyEncodedPostings(b)
}

p, err := decodeCachedPostings(b)
if err != nil {
    if strings.Contains(err.Error(), "unrecognize postings format") {
        // evict cache entry, fall back to fetching postings from index
        return fetchFromIndex(refs)
    }
    return err
}

When it happens

Trigger: Calling decodeCachedPostings with bytes lacking a recognized codecHeaderStreamedSnappy or diff-varint-snappy header: data written by an older Prometheus version, corrupt/truncated cache entries, or a manual/foreign encoder writing to the postings cache.

Common situations: Downgrading Prometheus so cached postings in the store cache use an unrecognized format; corrupted head/chunk data on disk after an unclean shutdown; manually migrating index-cache contents across versions.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• header not found
• got bad identifier
• mismatched checksum (got , expected )
• snappy decode
• postings

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/1433ce5eca0d93bd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/postings_codec.go:47

// and Varint is very efficient at encoding small values (values < 128 are encoded as
// single byte, values < 16384 are encoded as two bytes). Diff + varint reduces postings size
// significantly (to about 20% of original), snappy then halves it to ~10% of the original.

const (
	codecHeaderSnappy         = "dvs" // As in "diff+varint+snappy".
	codecHeaderStreamedSnappy = "dss" // As in "diffvarint+streamed snappy".
)

func decodePostings(input []byte) (closeablePostings, error) {
	var df func([]byte, bool) (closeablePostings, error)

	switch {
	case isDiffVarintSnappyEncodedPostings(input):
		df = diffVarintSnappyDecode
	case isDiffVarintSnappyStreamedEncodedPostings(input):
		df = diffVarintSnappyStreamedDecode
	default:
		return nil, fmt.Errorf("unrecognize postings format")
	}

	return df(input, false)
}

// isDiffVarintSnappyEncodedPostings returns true, if input looks like it has been encoded by diff+varint+snappy codec.
func isDiffVarintSnappyEncodedPostings(input []byte) bool {
	return bytes.HasPrefix(input, []byte(codecHeaderSnappy))
}

// isDiffVarintSnappyStreamedEncodedPostings returns true, if input looks like it has been encoded by diff+varint+snappy streamed codec.
func isDiffVarintSnappyStreamedEncodedPostings(input []byte) bool {
	return bytes.HasPrefix(input, []byte(codecHeaderStreamedSnappy))
}

// estimateSnappyStreamSize estimates the number of bytes
// needed for encoding length postings. Note that in reality
// the number of bytes needed could be much bigger if postings

View on GitHub (pinned to 35b8b99117)