thanos-io/thanos · error

series out of order; previous

Error message

series %v out of order; previous %v

What it means

The index requires series to be stored in sorted label-set order (that is how postings lookups work). GatherIndexHealthStats compares each series' label set with the previous one and errors with "series %v out of order; previous %v" when the order regresses. This indicates a corrupt or improperly written index, not a data problem in the chunks.

Solutions

1. Rebuild the block index with offline compaction (tsdb tooling) to restore sorted order
2. Repair or drop the block via block.Repair for known historical issues
3. Check Prometheus/Thanos versions that wrote the block and upgrade before re-writing data
4. Restore the block from a known-good backup

Example fix

// before
stats, err := block.GatherIndexHealthStats(ctx, bdir) // series {foo="bar"} out of order; previous {foo="baz"}
// after
// rebuild index via offline compaction
b, err := tsdb.OpenBlock(logger, bdir, nil, nil)
if err != nil { return err }
_, err = tsdb.CompactBlock(logger, bdir, b) // rewrites sorted index

meta, err := metadata.ReadFromDir(bdir)
if err != nil { return err }
if meta.PrometheusVersion() == "" || versionKnownBuggy(meta) {
    return fmt.Errorf("block %s written by suspect version %s", bdir, meta.Version)
}

stats, err := block.GatherIndexHealthStats(ctx, bdir)
if err != nil && strings.Contains(err.Error(), "out of order") {
    return rebuildIndexViaCompaction(logger, bdir)
}

When it happens

Trigger: GatherIndexHealthStats iterates postings and encounters series whose labels compare >= 0 against the previous series — from index corruption or indexes written by versions with sorting bugs (e.g. the tsdb issue fixed around symbols/sort ordering).

Common situations: Old blocks written by buggy Prometheus/Thanos versions, corrupted uploads, verification after migrating TSDB versions, blocks assembled by external tooling.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• input block index not valid
• get all postings
• label names
• metric label values
• read series

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fce7c5f7b4e5e0f4. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:286

		prevLset.CopyFrom(lset)

		id := p.At()
		if prevId != 0 {
			// Approximate size.
			seriesSize.Add(int64(id-prevId) * int64(offsetMultiplier))
		}
		prevId = id
		stats.TotalSeries++

		if err := r.Series(id, &builder, &chks); err != nil {
			return stats, errors.Wrap(err, "read series")
		}
		lset = builder.Labels()
		if lset.IsEmpty() {
			return stats, errors.Errorf("empty label set detected for series %d", id)
		}
		if !prevLset.IsEmpty() && labels.Compare(prevLset, lset) >= 0 {
			return stats, errors.Errorf("series %v out of order; previous %v", lset, prevLset)
		}
		var l0 *labels.Label
		lset.Range(func(l labels.Label) {
			if l0 != nil {
				if l.Name < l0.Name {
					stats.OutOfOrderLabels++
					level.Warn(logger).Log("msg",
						"out-of-order label set: known bug in Prometheus 2.8.0 and below",
						"labelset", lset.String(),
						"series", fmt.Sprintf("%d", id),
					)
				}
			}
			l0 = &l
		})

		if len(chks) == 0 {
			return stats, errors.Errorf("empty chunks for series %d", id)

View on GitHub (pinned to 35b8b99117)