thanos-io/thanos · critical

found chunks non-completely outside the block time range…

Error message

found %d chunks non-completely outside the block time range, found %d chunks completely outside the block time range

What it means

HealthStats.CriticalErr reports chunks found outside the block's declared [minTime, maxTime] range that are not auto-fixable: chunks partially overlapping the range (non-completely outside) and chunks entirely outside the range (complete outsiders, never accessed). Thanos treats these as critical block issues that can only be resolved by a manual repair procedure, since block metadata no longer matches actual chunk data.

Solutions

1. Run block.Repair with IgnoreCompleteOutsideChunk (plus IgnoreIssue347OutsideChunk/IgnoreDuplicateOutsideChunk) to rewrite the block dropping outsiders.
2. If only metadata boundaries are wrong, recompute and rewrite meta.json minTime/maxTime from actual chunk data, then re-verify.
3. Delete the corrupted block and re-upload/re-backfill it from source data.
4. Manually inspect with `thanos tools bucket inspect` to scope damage before repair; never edit meta.json by hand going forward.

Example fix

// before
return stats.AnyErr() // found 2 chunks non-completely outside..., found 1 chunks completely outside...
// after
_, err := block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource,
    block.IgnoreCompleteOutsideChunk, block.IgnoreIssue347OutsideChunk, block.IgnoreDuplicateOutsideChunk)
if err != nil { return err }
return block.VerifyIndex(ctx, logger, filepath.Join(dir, id.String(), "index"), minTime, maxTime)

stats, err := block.GatherIndexHealthStats(ctx, logger, indexFn, minTime, maxTime)
if err != nil { return err }
if stats.OutsideChunks > stats.Issue347OutsideChunks {
    return fmt.Errorf("block %s has %d non-recoverable outside chunks; manual repair needed", id, stats.OutsideChunks-stats.Issue347OutsideChunks)
}

if err := stats.CriticalErr(); err != nil {
    _, err := block.Repair(ctx, logger, dir, id, metadata.CompactorGrouperSource, block.IgnoreCompleteOutsideChunk)
    return err
}

When it happens

Trigger: GatherIndexHealthStats flags every chunk with c.MinTime < minTime || c.MaxTime > maxTime; those fully outside (c.MinTime > maxTime || c.MaxTime < minTime) increment CompleteOutsideChunks, the rest minus issue-347 chunks count as non-completely outside; CriticalErr returns this message when either count > 0, joined by AnyErr during VerifyIndex.

Common situations: Block metadata (meta.json minTime/maxTime) edited or truncated manually; blocks corrupted by interrupted compaction/upload; issues from old Thanos/Prometheus versions that miscalculated block boundaries; partial block downloads in store gateways.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• found chunks outside the block time range introduced by…
• open block
• / series have an average of %.3f out-of-order chunks: %.3f…
• joined health-check error messages (AnyErr)
• open index file

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/518e2c6386c11cdd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:148

	}
	return nil
}

// CriticalErr returns error if stats indicates critical block issue, that might solved only by manual repair procedure.
func (i HealthStats) CriticalErr() error {
	var errMsg []string

	n := i.OutsideChunks - (i.CompleteOutsideChunks + i.Issue347OutsideChunks)
	if n > 0 {
		errMsg = append(errMsg, fmt.Sprintf("found %d chunks non-completely outside the block time range", n))
	}

	if i.CompleteOutsideChunks > 0 {
		errMsg = append(errMsg, fmt.Sprintf("found %d chunks completely outside the block time range", i.CompleteOutsideChunks))
	}

	if len(errMsg) > 0 {
		return errors.New(strings.Join(errMsg, ", "))
	}

	return nil
}

// AnyErr returns error if stats indicates any block issue.
func (i HealthStats) AnyErr() error {
	var errMsg []string

	if err := i.CriticalErr(); err != nil {
		errMsg = append(errMsg, err.Error())
	}

	if err := i.Issue347OutsideChunksErr(); err != nil {
		errMsg = append(errMsg, err.Error())
	}

	if err := i.OutOfOrderLabelsErr(); err != nil {

View on GitHub (pinned to 35b8b99117)