thanos-io/thanos · warning

index contains postings with out of order labels

Error message

index contains %d postings with out of order labels

What it means

HealthStats.OutOfOrderLabelsErr reports that a TSDB block's index contains postings whose labels are stored out of order. This indicates the block was written by Prometheus <= 2.8.0, before the fix for Prometheus issue #5372; such blocks can cause incorrect or missing lookups in newer readers.

Solutions

1. Re-create the affected block: download it, re-run compaction/re-serialization (e.g. 'thanos tools bucket verify --repair' handles this by rewriting bad blocks).
2. Upgrade the producing Prometheus instances to > 2.8.0 so new blocks are healthy.
3. Identify affected blocks via the block health check/inspection tools and remove them from querier results until repaired.
4. If the block is unrecoverable and data is expendable, delete the block from the bucket after confirming the retention policy.

Example fix

// before: surfacing raw error and halting verification
return i.OutOfOrderLabelsErr()
// after: detect and let the planner repair legacy blocks
if err := i.OutOfOrderLabelsErr(); err != nil {
    // mark block for re-compaction (repair of Prometheus issue #5372)
    return block.IgnoreOrWrapErr(err)
}

// Go: inspect block health before relying on the index
stats, err := block.GatherIndexHealthStats(ctx, b, id, logger)
if err == nil && stats.OutOfOrderLabels > 0 {
    log.Warnf("block %s written by Prometheus <= 2.8.0; needs repair", id)
}

func hasOutOfOrderPostings(s block.HealthStats) bool { return s.OutOfOrderLabels > 0 }

if err := health.OutOfOrderLabelsErr(); err != nil {
    logger.Warnf("legacy block detected (Prometheus issue #5372): %v", err)
    // route block to repair/re-compaction instead of failing hard
    return nil
}

When it happens

Trigger: Calling OutOfOrderLabelsErr on a HealthStats (produced by block index health checks, e.g. 'thanos tools bucket verify' or index inspection) when HealthStats.OutOfOrderLabels > 0 during validation of an old block's index.

Common situations: Upgrading a Thanos/Prometheus deployment with historical blocks created by Prometheus 2.8.0 or older; running bucket verification on long-lived buckets containing legacy blocks.

Related errors

• add symbol
• next symbol
• postings
• series
• iterate series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b6780d479c991fa2. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:106

	ChunkMaxSize int64

	SeriesMinSize int64
	SeriesAvgSize int64
	SeriesMaxSize int64

	SingleSampleSeries int64
	SingleSampleChunks int64

	LabelNamesCount        int64
	MetricLabelValuesCount int64
}

// OutOfOrderLabelsErr returns an error if the HealthStats object indicates
// postings with out of order labels.  This is corrected by Prometheus Issue
// #5372 and affects Prometheus versions 2.8.0 and below.
func (i HealthStats) OutOfOrderLabelsErr() error {
	if i.OutOfOrderLabels > 0 {
		return errors.Errorf("index contains %d postings with out of order labels",
			i.OutOfOrderLabels)
	}
	return nil
}

// Issue347OutsideChunksErr returns error if stats indicates issue347 block issue, that is repaired explicitly before compaction (on plan block).
func (i HealthStats) Issue347OutsideChunksErr() error {
	if i.Issue347OutsideChunks > 0 {
		return errors.Errorf("found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347", i.Issue347OutsideChunks)
	}
	return nil
}

func (i HealthStats) OutOfOrderChunksErr() error {
	if i.OutOfOrderChunks > 0 {
		return errors.New(fmt.Sprintf(
			"%d/%d series have an average of %.3f out-of-order chunks: "+
				"%.3f of these are exact duplicates (in terms of data and time range)",

View on GitHub (pinned to 35b8b99117)