thanos-io/thanos · error

joined health-check error messages (AnyErr)

Error message

joined health-check error messages (AnyErr)

What it means

AnyErr is the aggregate health verdict for a block index: it joins the messages from CriticalErr, Issue347OutsideChunksErr, OutOfOrderLabelsErr and OutOfOrderChunksErr into a single comma-joined error. The message 'joined health-check error messages (AnyErr)' represents this aggregation; it is non-nil when any known block issue was detected by GatherIndexHealthStats. VerifyIndex uses it as its return value.

Solutions

1. Read the joined message to identify which sub-issue(s) apply (critical outsiders, issue 347, out-of-order labels, out-of-order chunks).
2. Run block.Repair with the appropriate ignore functions (IgnoreCompleteOutsideChunk, IgnoreIssue347OutsideChunk, IgnoreDuplicateOutsideChunk) to produce a fixed block.
3. For out-of-order labels (Prometheus <=2.8.0), upgrade Prometheus and regenerate/re-upload the block.
4. Mark/remove the bad block from the bucket and backfill from source if repair cannot fix it.

Example fix

// before
if err := stats.AnyErr(); err != nil { return errors.Wrap(err, "block is corrupted") }
// after: branch on sub-issues
if err := stats.CriticalErr(); err != nil { return repairBlock(ctx, dir, id) }
if err := stats.Issue347OutsideChunksErr(); err != nil { return repairWithIgnore(ctx, dir, id, block.IgnoreIssue347OutsideChunk) }
return nil

if err := block.VerifyIndex(ctx, logger, indexFn, minTime, maxTime); err != nil {
    logger.Warn("block index verification failed", "block", id, "err", err)
    // inspect joined message to decide repair strategy
    return markBlockForRepair(ctx, id, err)
}

When it happens

Trigger: Called at the end of block.VerifyIndex (stats.AnyErr()) after GatherIndexHealthStats walked the index; fires whenever any of: outside/critical chunks, issue-347 chunks, out-of-order labels (Prometheus <=2.8.0), or out-of-order chunks were found. Multiple issues appear joined as one message.

Common situations: `thanos tools bucket verify` failing on historical blocks; compaction planner marking blocks for repair; store gateway or querier operators investigating bad blocks in object storage; blocks produced by old Prometheus versions.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• open block
• found chunks outside the block time range introduced by…
• / series have an average of %.3f out-of-order chunks: %.3f…
• found chunks non-completely outside the block time range…
• open index file

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4dfe05ba9446a084. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:175

	if err := i.CriticalErr(); err != nil {
		errMsg = append(errMsg, err.Error())
	}

	if err := i.Issue347OutsideChunksErr(); err != nil {
		errMsg = append(errMsg, err.Error())
	}

	if err := i.OutOfOrderLabelsErr(); err != nil {
		errMsg = append(errMsg, err.Error())
	}

	if err := i.OutOfOrderChunksErr(); err != nil {
		errMsg = append(errMsg, err.Error())
	}

	if len(errMsg) > 0 {
		return errors.New(strings.Join(errMsg, ", "))
	}

	return nil
}

type minMaxSumInt64 struct {
	sum int64
	min int64
	max int64

	cnt int64
}

func newMinMaxSumInt64() minMaxSumInt64 {
	return minMaxSumInt64{
		min: math.MaxInt64,
		max: math.MinInt64,
	}

View on GitHub (pinned to 35b8b99117)