{"record":{"id":"c76c798b30cf0428","repo":"thanos-io/thanos","slug":"d-d-series-have-an-average-of-3f-out-of-order","errorCode":null,"errorMessage":"%d/%d series have an average of %.3f out-of-order chunks: %.3f of these are exact duplicates (in terms of data and time range)","messagePattern":"(.+?)/(.+?) series have an average of %\\.3f out-of-order chunks: %\\.3f of these are exact duplicates \\(in terms of data and time range\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/index.go","lineNumber":122,"sourceCode":"func (i HealthStats) OutOfOrderLabelsErr() error {\n\tif i.OutOfOrderLabels > 0 {\n\t\treturn errors.Errorf(\"index contains %d postings with out of order labels\",\n\t\t\ti.OutOfOrderLabels)\n\t}\n\treturn nil\n}\n\n// Issue347OutsideChunksErr returns error if stats indicates issue347 block issue, that is repaired explicitly before compaction (on plan block).\nfunc (i HealthStats) Issue347OutsideChunksErr() error {\n\tif i.Issue347OutsideChunks > 0 {\n\t\treturn errors.Errorf(\"found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347\", i.Issue347OutsideChunks)\n\t}\n\treturn nil\n}\n\nfunc (i HealthStats) OutOfOrderChunksErr() error {\n\tif i.OutOfOrderChunks > 0 {\n\t\treturn errors.New(fmt.Sprintf(\n\t\t\t\"%d/%d series have an average of %.3f out-of-order chunks: \"+\n\t\t\t\t\"%.3f of these are exact duplicates (in terms of data and time range)\",\n\t\t\ti.OutOfOrderSeries,\n\t\t\ti.TotalSeries,\n\t\t\tfloat64(i.OutOfOrderChunks)/float64(i.OutOfOrderSeries),\n\t\t\tfloat64(i.DuplicatedChunks)/float64(i.OutOfOrderChunks),\n\t\t))\n\t}\n\treturn nil\n}\n\n// CriticalErr returns error if stats indicates critical block issue, that might solved only by manual repair procedure.\nfunc (i HealthStats) CriticalErr() error {\n\tvar errMsg []string\n\n\tn := i.OutsideChunks - (i.CompleteOutsideChunks + i.Issue347OutsideChunks)\n\tif n > 0 {\n\t\terrMsg = append(errMsg, fmt.Sprintf(\"found %d chunks non-completely outside the block time range\", n))","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/index.go#L104-L140","documentation":"HealthStats.OutOfOrderChunksErr fires when the block index contains out-of-order chunks: chunks within a series whose time ranges overlap or go backwards. Thanos gathers these stats while walking all postings and reports the ratio of affected series and how many of the offending chunks are exact time-range duplicates. Such blocks indicate concurrent writes or duplicate ingestion and can break query correctness.","triggerScenarios":"GatherIndexHealthStats counts, per series, chunks i where c.MinTime <= previous chunk's MaxTime and the time ranges are not identical (ooo counter); OutOfOrderChunksErr returns this message when OutOfOrderChunks > 0, aggregated into AnyErr by VerifyIndex. Usually observed when Thanos compaction/verification scans blocks uploaded from multiple Prometheus replicas or from concurrent appends.","commonSituations":"Multiple Prometheus instances writing to the same block/store bucket; overlapping vertical sharding or deduplication misconfigurations; blocks produced during Prometheus crash/restart with concurrent head appends; data re-ingested from Kafka/backfill tooling.","solutions":["Repair the block using block.Repair with block.IgnoreDuplicateOutsideChunk to drop exact-duplicate chunks (errors if overlaps are not exact duplicates).","Enable Thanos compaction deduplication (compactor with proper replica labels) so overlapping blocks from replicas are resolved.","Re-create/re-upload the affected block from a single authoritative Prometheus data source.","Delete the corrupted block and backfill it correctly with thanos backfill/liquid tooling."],"exampleFix":"// before: verify fails with N/M series out-of-order\nstats, err := block.GatherIndexHealthStats(ctx, logger, indexFile, minTime, maxTime)\nif err := stats.AnyErr(); err != nil { return err }\n// after: repair with duplicate-ignore before verify\n_, err = block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource, block.IgnoreDuplicateOutsideChunk, block.IgnoreIssue347OutsideChunk)\nif err != nil { return err }","handlingStrategy":"validation","validationCode":"stats, err := block.GatherIndexHealthStats(ctx, logger, indexFn, minTime, maxTime)\nif err != nil { return err }\nif stats.OutOfOrderChunks > 0 {\n    return fmt.Errorf(\"block %s has %d out-of-order chunks; repair required\", id, stats.OutOfOrderChunks)\n}","typeGuard":null,"tryCatchPattern":"if err := block.VerifyIndex(ctx, logger, indexFn, minTime, maxTime); err != nil {\n    if strings.Contains(err.Error(), \"out-of-order chunks\") {\n        _, err = block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource, block.IgnoreDuplicateOutsideChunk)\n    }\n}","preventionTips":["Avoid multiple Prometheus replicas writing the same block without deduplication","Enable compactor deduplication with replica labels","Verify blocks before compaction"],"tags":["tsdb","out-of-order","thanos","block-index","duplicates"],"backgroundTag":"schema-validation-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}