thanos-io/thanos · critical · halt

block with not healthy index found

Error message

block with not healthy index found %s; Compaction level %v; Labels: %v

What it means

The block's index passed the gather step but stats.CriticalErr() found a critical health problem — the index is structurally broken (beyond reparable issues). Compaction is halted (halt error) because this block must not be compacted and the operator must intervene; it will not self-heal on retry.

Solutions

1. Run 'thanos tools bucket repair' (offline) which removes/replaces blocks with critical index issues
2. Identify the block ULID from logs and delete it if its source data can be re-uploaded from Prometheus
3. Check the origin TSDB data (sidecar/receive) and re-upload the block
4. Open an issue with Thanos/Prometheus if corruption is widespread — could be a known writer bug

Example fix

// before
// halting error, compactor exits on this block
# thanos tools bucket repair --objstore.bucket prom --data-dir /data
// after
// repair removes corrupt blocks; then re-run
# thanos compact --data-dir /data --objstore.bucket prom

// Detect critical index issues before compaction:
thanos tools bucket verify --objstore.bucket prom

When it happens

Trigger: GatherIndexHealthStats produced HealthStats whose CriticalErr() is non-nil: e.g. corrupt postings/symbols, index does not cover meta.MinTime..MaxTime, or other unrecoverable index corruption.

Common situations: Blocks corrupted by past bugs (Prometheus index writers), bad disks on the pushing sidecar, manual tampering with block files; known Thanos/Prometheus index corruption incidents requiring repair.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• block id , try running with --debug.accept-malformed-index
• iterate series
• repaired block is invalid
• gather index issues for block
• blocks with out-of-order chunks are dropped from compaction

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/6e5ef34a3e4bdba6. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1234

				if err := tracing.DoInSpanWithErr(ctx, "compaction_block_download", func(ctx context.Context) error {
					return block.Download(ctx, cg.logger, cg.bkt, meta.ULID, bdir, objstore.WithFetchConcurrency(cg.blockFilesConcurrency))
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return retry(errors.Wrapf(err, "download block %s", meta.ULID))
				}
				level.Debug(cg.logger).Log("msg", "downloaded block", "block", meta.ULID.String(), "duration", time.Since(start), "duration_ms", time.Since(start).Milliseconds())

				start = time.Now()
				// Ensure all input blocks are valid.
				var stats block.HealthStats
				if err := tracing.DoInSpanWithErr(ctx, "compaction_block_health_stats", func(ctx context.Context) (e error) {
					stats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)
					return e
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return errors.Wrapf(err, "gather index issues for block %s", bdir)
				}

				if err := stats.CriticalErr(); err != nil {
					return halt(errors.Wrapf(err, "block with not healthy index found %s; Compaction level %v; Labels: %v", bdir, meta.Compaction.Level, meta.Thanos.Labels))
				}

				if err := stats.OutOfOrderChunksErr(); err != nil {
					return outOfOrderChunkError(errors.Wrapf(err, "blocks with out-of-order chunks are dropped from compaction:  %s", bdir), meta.ULID)
				}

				if err := stats.Issue347OutsideChunksErr(); err != nil {
					return issue347Error(errors.Wrapf(err, "invalid, but reparable block %s", bdir), meta.ULID)
				}

				if err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {
					return errors.Wrapf(err,
						"block id %s, try running with --debug.accept-malformed-index", meta.ULID)
				}
				level.Debug(cg.logger).Log("msg", "verified block", "block", meta.ULID.String(), "duration", time.Since(start), "duration_ms", time.Since(start).Milliseconds())
				return nil
			})
		}(errCtx, m)

View on GitHub (pinned to 35b8b99117)