{"record":{"id":"f6143aa239b5f957","repo":"thanos-io/thanos","slug":"blocks-with-out-of-order-chunks-are-dropped-from-c","errorCode":null,"errorMessage":"blocks with out-of-order chunks are dropped from compaction:  %s","messagePattern":"blocks with out-of-order chunks are dropped from compaction:  (.+?)","errorType":"exception","errorClass":"outOfOrderChunkError","httpStatus":null,"severity":"warning","filePath":"pkg/compact/compact.go","lineNumber":1238,"sourceCode":"\t\t\t\t}\n\t\t\t\tlevel.Debug(cg.logger).Log(\"msg\", \"downloaded block\", \"block\", meta.ULID.String(), \"duration\", time.Since(start), \"duration_ms\", time.Since(start).Milliseconds())\n\n\t\t\t\tstart = time.Now()\n\t\t\t\t// Ensure all input blocks are valid.\n\t\t\t\tvar stats block.HealthStats\n\t\t\t\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_block_health_stats\", func(ctx context.Context) (e error) {\n\t\t\t\t\tstats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)\n\t\t\t\t\treturn e\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn errors.Wrapf(err, \"gather index issues for block %s\", bdir)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.CriticalErr(); err != nil {\n\t\t\t\t\treturn halt(errors.Wrapf(err, \"block with not healthy index found %s; Compaction level %v; Labels: %v\", bdir, meta.Compaction.Level, meta.Thanos.Labels))\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderChunksErr(); err != nil {\n\t\t\t\t\treturn outOfOrderChunkError(errors.Wrapf(err, \"blocks with out-of-order chunks are dropped from compaction:  %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.Issue347OutsideChunksErr(); err != nil {\n\t\t\t\t\treturn issue347Error(errors.Wrapf(err, \"invalid, but reparable block %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {\n\t\t\t\t\treturn errors.Wrapf(err,\n\t\t\t\t\t\t\"block id %s, try running with --debug.accept-malformed-index\", meta.ULID)\n\t\t\t\t}\n\t\t\t\tlevel.Debug(cg.logger).Log(\"msg\", \"verified block\", \"block\", meta.ULID.String(), \"duration\", time.Since(start), \"duration_ms\", time.Since(start).Milliseconds())\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}(errCtx, m)\n\n\t\ttoCompactDirs = append(toCompactDirs, bdir)\n\t}\n\tsourceBlockStr := fmt.Sprintf(\"%v\", toCompactDirs)","sourceCodeStart":1220,"sourceCodeEnd":1256,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1220-L1256","documentation":"The block's index contains out-of-order chunks, a condition caused by historical Prometheus/Thanos bugs. Such blocks are deliberately excluded from compaction: Thanos returns a typed outOfOrderChunkError carrying the block ULID so the syncer can ignore (and later optionally delete) the block instead of producing wrong compacted data.","triggerScenarios":"stats.OutOfOrderChunksErr() is non-nil after index health gathering — chunk references in the index are not in strictly increasing time order. The message is informational-ish: the block is dropped from compaction, not a crash.","commonSituations":"Very old blocks created by Prometheus versions affected by the out-of-order-chunks bug (fixed in Prometheus 2.x releases); ingesting samples with clock skew historically written out of order.","solutions":["Let the compactor exclude the block — this is expected handling; verify with 'thanos tools bucket inspect'","Run 'thanos tools bucket verify --repair' to remove known-bad blocks from the bucket","Delete the affected block if the data is expendable; else re-create from source TSDB with an up-to-date Prometheus","Ensure Prometheus/sidecar versions are current so new blocks aren't affected"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Find blocks affected by out-of-order chunks:\nthanos tools bucket verify --objstore.bucket prom | grep -i 'out-of-order'","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"out-of-order chunks\") {\n    log.Warn(\"block dropped from compaction (expected for legacy blocks)\")\n}","preventionTips":["Migrate legacy buckets through bucket repair once","Run current Prometheus versions for all data producers","Treat this as a warning, not an outage — blocks are excluded safely"],"tags":["compaction","index","out-of-order","thanos"],"backgroundTag":"schema-validation-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}