{"record":{"id":"6e5ef34a3e4bdba6","repo":"thanos-io/thanos","slug":"block-with-not-healthy-index-found-s-compaction","errorCode":null,"errorMessage":"block with not healthy index found %s; Compaction level %v; Labels: %v","messagePattern":"block with not healthy index found (.+?); Compaction level (.+?); Labels: (.+?)","errorType":"exception","errorClass":"halt","httpStatus":null,"severity":"critical","filePath":"pkg/compact/compact.go","lineNumber":1234,"sourceCode":"\t\t\t\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_block_download\", func(ctx context.Context) error {\n\t\t\t\t\treturn block.Download(ctx, cg.logger, cg.bkt, meta.ULID, bdir, objstore.WithFetchConcurrency(cg.blockFilesConcurrency))\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn retry(errors.Wrapf(err, \"download block %s\", meta.ULID))\n\t\t\t\t}\n\t\t\t\tlevel.Debug(cg.logger).Log(\"msg\", \"downloaded block\", \"block\", meta.ULID.String(), \"duration\", time.Since(start), \"duration_ms\", time.Since(start).Milliseconds())\n\n\t\t\t\tstart = time.Now()\n\t\t\t\t// Ensure all input blocks are valid.\n\t\t\t\tvar stats block.HealthStats\n\t\t\t\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_block_health_stats\", func(ctx context.Context) (e error) {\n\t\t\t\t\tstats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)\n\t\t\t\t\treturn e\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn errors.Wrapf(err, \"gather index issues for block %s\", bdir)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.CriticalErr(); err != nil {\n\t\t\t\t\treturn halt(errors.Wrapf(err, \"block with not healthy index found %s; Compaction level %v; Labels: %v\", bdir, meta.Compaction.Level, meta.Thanos.Labels))\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderChunksErr(); err != nil {\n\t\t\t\t\treturn outOfOrderChunkError(errors.Wrapf(err, \"blocks with out-of-order chunks are dropped from compaction:  %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.Issue347OutsideChunksErr(); err != nil {\n\t\t\t\t\treturn issue347Error(errors.Wrapf(err, \"invalid, but reparable block %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {\n\t\t\t\t\treturn errors.Wrapf(err,\n\t\t\t\t\t\t\"block id %s, try running with --debug.accept-malformed-index\", meta.ULID)\n\t\t\t\t}\n\t\t\t\tlevel.Debug(cg.logger).Log(\"msg\", \"verified block\", \"block\", meta.ULID.String(), \"duration\", time.Since(start), \"duration_ms\", time.Since(start).Milliseconds())\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}(errCtx, m)","sourceCodeStart":1216,"sourceCodeEnd":1252,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1216-L1252","documentation":"The block's index passed the gather step but stats.CriticalErr() found a critical health problem — the index is structurally broken (beyond reparable issues). Compaction is halted (halt error) because this block must not be compacted and the operator must intervene; it will not self-heal on retry.","triggerScenarios":"GatherIndexHealthStats produced HealthStats whose CriticalErr() is non-nil: e.g. corrupt postings/symbols, index does not cover meta.MinTime..MaxTime, or other unrecoverable index corruption.","commonSituations":"Blocks corrupted by past bugs (Prometheus index writers), bad disks on the pushing sidecar, manual tampering with block files; known Thanos/Prometheus index corruption incidents requiring repair.","solutions":["Run 'thanos tools bucket repair' (offline) which removes/replaces blocks with critical index issues","Identify the block ULID from logs and delete it if its source data can be re-uploaded from Prometheus","Check the origin TSDB data (sidecar/receive) and re-upload the block","Open an issue with Thanos/Prometheus if corruption is widespread — could be a known writer bug"],"exampleFix":"// before\n// halting error, compactor exits on this block\n# thanos tools bucket repair --objstore.bucket prom --data-dir /data\n// after\n// repair removes corrupt blocks; then re-run\n# thanos compact --data-dir /data --objstore.bucket prom","handlingStrategy":"validation","validationCode":"// Detect critical index issues before compaction:\nthanos tools bucket verify --objstore.bucket prom","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Schedule regular bucket verify + repair windows","Keep Prometheus/sidecar versions updated to avoid known index writer bugs","Alert on halt-level compaction errors — they need manual intervention"],"tags":["compaction","index","corruption","thanos"],"backgroundTag":"checksum-mismatch","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}