{"record":{"id":"b6780d479c991fa2","repo":"thanos-io/thanos","slug":"index-contains-d-postings-with-out-of-order-label","errorCode":null,"errorMessage":"index contains %d postings with out of order labels","messagePattern":"index contains (.+?) postings with out of order labels","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/block/index.go","lineNumber":106,"sourceCode":"\tChunkMaxSize int64\n\n\tSeriesMinSize int64\n\tSeriesAvgSize int64\n\tSeriesMaxSize int64\n\n\tSingleSampleSeries int64\n\tSingleSampleChunks int64\n\n\tLabelNamesCount        int64\n\tMetricLabelValuesCount int64\n}\n\n// OutOfOrderLabelsErr returns an error if the HealthStats object indicates\n// postings with out of order labels.  This is corrected by Prometheus Issue\n// #5372 and affects Prometheus versions 2.8.0 and below.\nfunc (i HealthStats) OutOfOrderLabelsErr() error {\n\tif i.OutOfOrderLabels > 0 {\n\t\treturn errors.Errorf(\"index contains %d postings with out of order labels\",\n\t\t\ti.OutOfOrderLabels)\n\t}\n\treturn nil\n}\n\n// Issue347OutsideChunksErr returns error if stats indicates issue347 block issue, that is repaired explicitly before compaction (on plan block).\nfunc (i HealthStats) Issue347OutsideChunksErr() error {\n\tif i.Issue347OutsideChunks > 0 {\n\t\treturn errors.Errorf(\"found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347\", i.Issue347OutsideChunks)\n\t}\n\treturn nil\n}\n\nfunc (i HealthStats) OutOfOrderChunksErr() error {\n\tif i.OutOfOrderChunks > 0 {\n\t\treturn errors.New(fmt.Sprintf(\n\t\t\t\"%d/%d series have an average of %.3f out-of-order chunks: \"+\n\t\t\t\t\"%.3f of these are exact duplicates (in terms of data and time range)\",","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/index.go#L88-L124","documentation":"HealthStats.OutOfOrderLabelsErr reports that a TSDB block's index contains postings whose labels are stored out of order. This indicates the block was written by Prometheus <= 2.8.0, before the fix for Prometheus issue #5372; such blocks can cause incorrect or missing lookups in newer readers.","triggerScenarios":"Calling OutOfOrderLabelsErr on a HealthStats (produced by block index health checks, e.g. 'thanos tools bucket verify' or index inspection) when HealthStats.OutOfOrderLabels > 0 during validation of an old block's index.","commonSituations":"Upgrading a Thanos/Prometheus deployment with historical blocks created by Prometheus 2.8.0 or older; running bucket verification on long-lived buckets containing legacy blocks.","solutions":["Re-create the affected block: download it, re-run compaction/re-serialization (e.g. 'thanos tools bucket verify --repair' handles this by rewriting bad blocks).","Upgrade the producing Prometheus instances to > 2.8.0 so new blocks are healthy.","Identify affected blocks via the block health check/inspection tools and remove them from querier results until repaired.","If the block is unrecoverable and data is expendable, delete the block from the bucket after confirming the retention policy."],"exampleFix":"// before: surfacing raw error and halting verification\nreturn i.OutOfOrderLabelsErr()\n// after: detect and let the planner repair legacy blocks\nif err := i.OutOfOrderLabelsErr(); err != nil {\n    // mark block for re-compaction (repair of Prometheus issue #5372)\n    return block.IgnoreOrWrapErr(err)\n}","handlingStrategy":"try-catch","validationCode":"// Go: inspect block health before relying on the index\nstats, err := block.GatherIndexHealthStats(ctx, b, id, logger)\nif err == nil && stats.OutOfOrderLabels > 0 {\n    log.Warnf(\"block %s written by Prometheus <= 2.8.0; needs repair\", id)\n}","typeGuard":"func hasOutOfOrderPostings(s block.HealthStats) bool { return s.OutOfOrderLabels > 0 }","tryCatchPattern":"if err := health.OutOfOrderLabelsErr(); err != nil {\n    logger.Warnf(\"legacy block detected (Prometheus issue #5372): %v\", err)\n    // route block to repair/re-compaction instead of failing hard\n    return nil\n}","preventionTips":["Keep Prometheus writers above v2.8.0 so new blocks never contain out-of-order postings.","Run periodic bucket verification to detect legacy blocks.","Plan repair/re-compaction for blocks created before upgrading.","Track the Prometheus version that produced each block in your metadata."],"tags":["tsdb","index","block-health","prometheus-compat"],"backgroundTag":"unexpected-response-shape","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}