/ series have an average of %.3f out-of-order chunks: %.3f…

Error message

%d/%d series have an average of %.3f out-of-order chunks: %.3f of these are exact duplicates (in terms of data and time range)

What it means

HealthStats.OutOfOrderChunksErr fires when the block index contains out-of-order chunks: chunks within a series whose time ranges overlap or go backwards. Thanos gathers these stats while walking all postings and reports the ratio of affected series and how many of the offending chunks are exact time-range duplicates. Such blocks indicate concurrent writes or duplicate ingestion and can break query correctness.

Solutions

Repair the block using block.Repair with block.IgnoreDuplicateOutsideChunk to drop exact-duplicate chunks (errors if overlaps are not exact duplicates).
Enable Thanos compaction deduplication (compactor with proper replica labels) so overlapping blocks from replicas are resolved.
Re-create/re-upload the affected block from a single authoritative Prometheus data source.
Delete the corrupted block and backfill it correctly with thanos backfill/liquid tooling.

Example fix

// before: verify fails with N/M series out-of-order
stats, err := block.GatherIndexHealthStats(ctx, logger, indexFile, minTime, maxTime)
if err := stats.AnyErr(); err != nil { return err }
// after: repair with duplicate-ignore before verify
_, err = block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource, block.IgnoreDuplicateOutsideChunk, block.IgnoreIssue347OutsideChunk)
if err != nil { return err }

Defensive patterns

Strategy: validation

Validate before calling

stats, err := block.GatherIndexHealthStats(ctx, logger, indexFn, minTime, maxTime)
if err != nil { return err }
if stats.OutOfOrderChunks > 0 {
    return fmt.Errorf("block %s has %d out-of-order chunks; repair required", id, stats.OutOfOrderChunks)
}

Try / catch

if err := block.VerifyIndex(ctx, logger, indexFn, minTime, maxTime); err != nil {
    if strings.Contains(err.Error(), "out-of-order chunks") {
        _, err = block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource, block.IgnoreDuplicateOutsideChunk)
    }
}

Prevention

Avoid multiple Prometheus replicas writing the same block without deduplication
Enable compactor deduplication with replica labels
Verify blocks before compaction

When it happens

Trigger: GatherIndexHealthStats counts, per series, chunks i where c.MinTime <= previous chunk's MaxTime and the time ranges are not identical (ooo counter); OutOfOrderChunksErr returns this message when OutOfOrderChunks > 0, aggregated into AnyErr by VerifyIndex. Usually observed when Thanos compaction/verification scans blocks uploaded from multiple Prometheus replicas or from concurrent appends.

Common situations: Multiple Prometheus instances writing to the same block/store bucket; overlapping vertical sharding or deduplication misconfigurations; blocks produced during Prometheus crash/restart with concurrent head appends; data re-ingested from Kafka/backfill tooling.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/c76c798b30cf0428. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:122

func (i HealthStats) OutOfOrderLabelsErr() error {
	if i.OutOfOrderLabels > 0 {
		return errors.Errorf("index contains %d postings with out of order labels",
			i.OutOfOrderLabels)
	}
	return nil
}

// Issue347OutsideChunksErr returns error if stats indicates issue347 block issue, that is repaired explicitly before compaction (on plan block).
func (i HealthStats) Issue347OutsideChunksErr() error {
	if i.Issue347OutsideChunks > 0 {
		return errors.Errorf("found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347", i.Issue347OutsideChunks)
	}
	return nil
}

func (i HealthStats) OutOfOrderChunksErr() error {
	if i.OutOfOrderChunks > 0 {
		return errors.New(fmt.Sprintf(
			"%d/%d series have an average of %.3f out-of-order chunks: "+
				"%.3f of these are exact duplicates (in terms of data and time range)",
			i.OutOfOrderSeries,
			i.TotalSeries,
			float64(i.OutOfOrderChunks)/float64(i.OutOfOrderSeries),
			float64(i.DuplicatedChunks)/float64(i.OutOfOrderChunks),
		))
	}
	return nil
}

// CriticalErr returns error if stats indicates critical block issue, that might solved only by manual repair procedure.
func (i HealthStats) CriticalErr() error {
	var errMsg []string

	n := i.OutsideChunks - (i.CompleteOutsideChunks + i.Issue347OutsideChunks)
	if n > 0 {
		errMsg = append(errMsg, fmt.Sprintf("found %d chunks non-completely outside the block time range", n))

View on GitHub (pinned to 35b8b99117)