thanos-io/thanos · warning · outOfOrderChunkError

blocks with out-of-order chunks are dropped from compaction

Error message

blocks with out-of-order chunks are dropped from compaction:  %s

What it means

The block's index contains out-of-order chunks, a condition caused by historical Prometheus/Thanos bugs. Such blocks are deliberately excluded from compaction: Thanos returns a typed outOfOrderChunkError carrying the block ULID so the syncer can ignore (and later optionally delete) the block instead of producing wrong compacted data.

Solutions

1. Let the compactor exclude the block — this is expected handling; verify with 'thanos tools bucket inspect'
2. Run 'thanos tools bucket verify --repair' to remove known-bad blocks from the bucket
3. Delete the affected block if the data is expendable; else re-create from source TSDB with an up-to-date Prometheus
4. Ensure Prometheus/sidecar versions are current so new blocks aren't affected

// Find blocks affected by out-of-order chunks:
thanos tools bucket verify --objstore.bucket prom | grep -i 'out-of-order'

if err != nil && strings.Contains(err.Error(), "out-of-order chunks") {
    log.Warn("block dropped from compaction (expected for legacy blocks)")
}

When it happens

Trigger: stats.OutOfOrderChunksErr() is non-nil after index health gathering — chunk references in the index are not in strictly increasing time order. The message is informational-ish: the block is dropped from compaction, not a crash.

Common situations: Very old blocks created by Prometheus versions affected by the out-of-order-chunks bug (fixed in Prometheus 2.x releases); ingesting samples with clock skew historically written out of order.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• gather index issues for block
• block with not healthy index found
• invalid, but reparable block
• block id , try running with --debug.accept-malformed-index
• create bucket compactor

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f6143aa239b5f957. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1238

				}
				level.Debug(cg.logger).Log("msg", "downloaded block", "block", meta.ULID.String(), "duration", time.Since(start), "duration_ms", time.Since(start).Milliseconds())

				start = time.Now()
				// Ensure all input blocks are valid.
				var stats block.HealthStats
				if err := tracing.DoInSpanWithErr(ctx, "compaction_block_health_stats", func(ctx context.Context) (e error) {
					stats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)
					return e
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return errors.Wrapf(err, "gather index issues for block %s", bdir)
				}

				if err := stats.CriticalErr(); err != nil {
					return halt(errors.Wrapf(err, "block with not healthy index found %s; Compaction level %v; Labels: %v", bdir, meta.Compaction.Level, meta.Thanos.Labels))
				}

				if err := stats.OutOfOrderChunksErr(); err != nil {
					return outOfOrderChunkError(errors.Wrapf(err, "blocks with out-of-order chunks are dropped from compaction:  %s", bdir), meta.ULID)
				}

				if err := stats.Issue347OutsideChunksErr(); err != nil {
					return issue347Error(errors.Wrapf(err, "invalid, but reparable block %s", bdir), meta.ULID)
				}

				if err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {
					return errors.Wrapf(err,
						"block id %s, try running with --debug.accept-malformed-index", meta.ULID)
				}
				level.Debug(cg.logger).Log("msg", "verified block", "block", meta.ULID.String(), "duration", time.Since(start), "duration_ms", time.Since(start).Milliseconds())
				return nil
			})
		}(errCtx, m)

		toCompactDirs = append(toCompactDirs, bdir)
	}
	sourceBlockStr := fmt.Sprintf("%v", toCompactDirs)

View on GitHub (pinned to 35b8b99117)