found chunks outside the block time range introduced by…

Error message

found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347

What it means

HealthStats.Issue347OutsideChunksErr reports chunks found in a TSDB block index whose minTime equals the block's maxTime, an artifact of the known Prometheus TSDB bug https://github.com/prometheus/tsdb/issues/347. Thanos throws this error during index verification to flag blocks containing such chunks so they can be repaired (dropped) explicitly before compaction via block planning. The chunks are considered safe to delete because they are duplicated across two blocks.

Solutions

Run the Thanos block repair flow (block Repair / compaction planner) which uses IgnoreIssue347OutsideChunk to drop these chunks and rewrite the block.
Re-compact the affected block after repair so the new index no longer contains chunks at minTime == maxTime.
If only affected blocks keep failing, re-upload or re-create the block from raw Prometheus data with a fixed Prometheus version.
Verify with `thanos tools bucket verify` after repair to confirm the stats are clean.

Example fix

// before: verify fails
if err := stats.AnyErr(); err != nil { return err } // found 3 chunks outside ... issue 347
// after: run repair before verify
repairedID, err := block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource, block.IgnoreIssue347OutsideChunk)
if err != nil { return err }
return block.VerifyIndex(ctx, logger, filepath.Join(dir, repairedID.String(), "index"), minTime, maxTime)

Defensive patterns

Strategy: validation

Validate before calling

stats, err := block.GatherIndexHealthStats(ctx, logger, indexFn, minTime, maxTime)
if err != nil { return err }
if stats.Issue347OutsideChunks > 0 {
    // schedule repair before compaction instead of failing
    return repairIssue347(ctx, dir, id)
}

Try / catch

if err := stats.AnyErr(); err != nil {
    if strings.Contains(err.Error(), "issues/347") {
        // repair path
    }
}

Prevention

Run Thanos compactor's repair flow regularly on old blocks
Use a Prometheus version with TSDB issue 347 fixed
Verify blocks after upload to object storage

When it happens

Trigger: GatherIndexHealthStats scans all series/chunks and counts chunks where c.MinTime == block.maxTime (and the chunk is otherwise outside the block range); Issue347OutsideChunksErr (via AnyErr, used by VerifyIndex) then fires whenever Issue347OutsideChunks > 0. Also surfaced during Thanos compaction planning when a block plan runs the repair flow.

Common situations: Blocks produced by older Prometheus/Thanos versions affected by TSDB issue 347 (head chunks written at compaction boundary); verifying or compacting historical blocks uploaded before the bug was fixed; running `thanos tools bucket verify` on old store data.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/ec2855dae0b30012. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:115

	LabelNamesCount        int64
	MetricLabelValuesCount int64
}

// OutOfOrderLabelsErr returns an error if the HealthStats object indicates
// postings with out of order labels.  This is corrected by Prometheus Issue
// #5372 and affects Prometheus versions 2.8.0 and below.
func (i HealthStats) OutOfOrderLabelsErr() error {
	if i.OutOfOrderLabels > 0 {
		return errors.Errorf("index contains %d postings with out of order labels",
			i.OutOfOrderLabels)
	}
	return nil
}

// Issue347OutsideChunksErr returns error if stats indicates issue347 block issue, that is repaired explicitly before compaction (on plan block).
func (i HealthStats) Issue347OutsideChunksErr() error {
	if i.Issue347OutsideChunks > 0 {
		return errors.Errorf("found %d chunks outside the block time range introduced by https://github.com/prometheus/tsdb/issues/347", i.Issue347OutsideChunks)
	}
	return nil
}

func (i HealthStats) OutOfOrderChunksErr() error {
	if i.OutOfOrderChunks > 0 {
		return errors.New(fmt.Sprintf(
			"%d/%d series have an average of %.3f out-of-order chunks: "+
				"%.3f of these are exact duplicates (in terms of data and time range)",
			i.OutOfOrderSeries,
			i.TotalSeries,
			float64(i.OutOfOrderChunks)/float64(i.OutOfOrderSeries),
			float64(i.DuplicatedChunks)/float64(i.OutOfOrderChunks),
		))
	}
	return nil
}

View on GitHub (pinned to 35b8b99117)