{"record":{"id":"ec2855dae0b30012","repo":"thanos-io/thanos","slug":"found-d-chunks-outside-the-block-time-range-intro","errorCode":null,"errorMessage":"found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347","messagePattern":"found (.+?) chunks outside the block time range introduced by https://github\\.com/prometheus/tsdb/issues/347","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/block/index.go","lineNumber":115,"sourceCode":"\tLabelNamesCount        int64\n\tMetricLabelValuesCount int64\n}\n\n// OutOfOrderLabelsErr returns an error if the HealthStats object indicates\n// postings with out of order labels.  This is corrected by Prometheus Issue\n// #5372 and affects Prometheus versions 2.8.0 and below.\nfunc (i HealthStats) OutOfOrderLabelsErr() error {\n\tif i.OutOfOrderLabels > 0 {\n\t\treturn errors.Errorf(\"index contains %d postings with out of order labels\",\n\t\t\ti.OutOfOrderLabels)\n\t}\n\treturn nil\n}\n\n// Issue347OutsideChunksErr returns error if stats indicates issue347 block issue, that is repaired explicitly before compaction (on plan block).\nfunc (i HealthStats) Issue347OutsideChunksErr() error {\n\tif i.Issue347OutsideChunks > 0 {\n\t\treturn errors.Errorf(\"found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347\", i.Issue347OutsideChunks)\n\t}\n\treturn nil\n}\n\nfunc (i HealthStats) OutOfOrderChunksErr() error {\n\tif i.OutOfOrderChunks > 0 {\n\t\treturn errors.New(fmt.Sprintf(\n\t\t\t\"%d/%d series have an average of %.3f out-of-order chunks: \"+\n\t\t\t\t\"%.3f of these are exact duplicates (in terms of data and time range)\",\n\t\t\ti.OutOfOrderSeries,\n\t\t\ti.TotalSeries,\n\t\t\tfloat64(i.OutOfOrderChunks)/float64(i.OutOfOrderSeries),\n\t\t\tfloat64(i.DuplicatedChunks)/float64(i.OutOfOrderChunks),\n\t\t))\n\t}\n\treturn nil\n}\n","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/index.go#L97-L133","documentation":"HealthStats.Issue347OutsideChunksErr reports chunks found in a TSDB block index whose minTime equals the block's maxTime, an artifact of the known Prometheus TSDB bug https://github.com/prometheus/tsdb/issues/347. Thanos throws this error during index verification to flag blocks containing such chunks so they can be repaired (dropped) explicitly before compaction via block planning. The chunks are considered safe to delete because they are duplicated across two blocks.","triggerScenarios":"GatherIndexHealthStats scans all series/chunks and counts chunks where c.MinTime == block.maxTime (and the chunk is otherwise outside the block range); Issue347OutsideChunksErr (via AnyErr, used by VerifyIndex) then fires whenever Issue347OutsideChunks > 0. Also surfaced during Thanos compaction planning when a block plan runs the repair flow.","commonSituations":"Blocks produced by older Prometheus/Thanos versions affected by TSDB issue 347 (head chunks written at compaction boundary); verifying or compacting historical blocks uploaded before the bug was fixed; running `thanos tools bucket verify` on old store data.","solutions":["Run the Thanos block repair flow (block Repair / compaction planner) which uses IgnoreIssue347OutsideChunk to drop these chunks and rewrite the block.","Re-compact the affected block after repair so the new index no longer contains chunks at minTime == maxTime.","If only affected blocks keep failing, re-upload or re-create the block from raw Prometheus data with a fixed Prometheus version.","Verify with `thanos tools bucket verify` after repair to confirm the stats are clean."],"exampleFix":"// before: verify fails\nif err := stats.AnyErr(); err != nil { return err } // found 3 chunks outside ... issue 347\n// after: run repair before verify\nrepairedID, err := block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource, block.IgnoreIssue347OutsideChunk)\nif err != nil { return err }\nreturn block.VerifyIndex(ctx, logger, filepath.Join(dir, repairedID.String(), \"index\"), minTime, maxTime)","handlingStrategy":"validation","validationCode":"stats, err := block.GatherIndexHealthStats(ctx, logger, indexFn, minTime, maxTime)\nif err != nil { return err }\nif stats.Issue347OutsideChunks > 0 {\n    // schedule repair before compaction instead of failing\n    return repairIssue347(ctx, dir, id)\n}","typeGuard":null,"tryCatchPattern":"if err := stats.AnyErr(); err != nil {\n    if strings.Contains(err.Error(), \"issues/347\") {\n        // repair path\n    }\n}","preventionTips":["Run Thanos compactor's repair flow regularly on old blocks","Use a Prometheus version with TSDB issue 347 fixed","Verify blocks after upload to object storage"],"tags":["tsdb","block-index","thanos","compaction","data-corruption"],"backgroundTag":"checksum-mismatch","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}