{"record":{"id":"1586d21b6da7c3ca","repo":"thanos-io/thanos","slug":"invalid-but-reparable-block-s","errorCode":null,"errorMessage":"invalid, but reparable block %s","messagePattern":"invalid, but reparable block (.+?)","errorType":"exception","errorClass":"issue347Error","httpStatus":null,"severity":"warning","filePath":"pkg/compact/compact.go","lineNumber":1242,"sourceCode":"\t\t\t\t// Ensure all input blocks are valid.\n\t\t\t\tvar stats block.HealthStats\n\t\t\t\tif err := tracing.DoInSpanWithErr(ctx, \"compaction_block_health_stats\", func(ctx context.Context) (e error) {\n\t\t\t\t\tstats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)\n\t\t\t\t\treturn e\n\t\t\t\t}, opentracing.Tags{\"block.id\": meta.ULID}); err != nil {\n\t\t\t\t\treturn errors.Wrapf(err, \"gather index issues for block %s\", bdir)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.CriticalErr(); err != nil {\n\t\t\t\t\treturn halt(errors.Wrapf(err, \"block with not healthy index found %s; Compaction level %v; Labels: %v\", bdir, meta.Compaction.Level, meta.Thanos.Labels))\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderChunksErr(); err != nil {\n\t\t\t\t\treturn outOfOrderChunkError(errors.Wrapf(err, \"blocks with out-of-order chunks are dropped from compaction:  %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.Issue347OutsideChunksErr(); err != nil {\n\t\t\t\t\treturn issue347Error(errors.Wrapf(err, \"invalid, but reparable block %s\", bdir), meta.ULID)\n\t\t\t\t}\n\n\t\t\t\tif err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {\n\t\t\t\t\treturn errors.Wrapf(err,\n\t\t\t\t\t\t\"block id %s, try running with --debug.accept-malformed-index\", meta.ULID)\n\t\t\t\t}\n\t\t\t\tlevel.Debug(cg.logger).Log(\"msg\", \"verified block\", \"block\", meta.ULID.String(), \"duration\", time.Since(start), \"duration_ms\", time.Since(start).Milliseconds())\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}(errCtx, m)\n\n\t\ttoCompactDirs = append(toCompactDirs, bdir)\n\t}\n\tsourceBlockStr := fmt.Sprintf(\"%v\", toCompactDirs)\n\n\tif err := g.Wait(); err != nil {\n\t\treturn false, nil, err\n\t}","sourceCodeStart":1224,"sourceCodeEnd":1260,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1224-L1260","documentation":"The block is affected by the 'issue 347' bug (chunks referenced in the index fall outside the block's time range). Such a block is invalid but reparable: Thanos wraps it in a typed issue347Error with the ULID so the syncer can later repair the block's meta/index instead of failing compaction outright.","triggerScenarios":"stats.Issue347OutsideChunksErr() is non-nil during index health validation of a downloaded block — index references chunks outside [MinTime, MaxTime].","commonSituations":"Blocks produced by Prometheus versions affected by issue #347; old buckets that never ran the repair tool; migration of legacy data.","solutions":["Run 'thanos tools bucket verify --repair' to repair blocks affected by issue 347","Or delete the block and re-upload from source Prometheus if repair is not possible","Keep Thanos/Prometheus updated; blocks produced by fixed writers won't exhibit this"],"exampleFix":"// before\n# thanos tools bucket verify --objstore.bucket prom // reports issue347 blocks\n// after\n# thanos tools bucket verify --repair --objstore.bucket prom","handlingStrategy":"validation","validationCode":"// Detect issue-347 blocks before compaction:\nthanos tools bucket verify --objstore.bucket prom | grep -i issue347","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run 'thanos tools bucket verify --repair' after migrating old buckets","Keep writer (Prometheus) versions up to date","Let the syncer auto-repair: ensure repair compatibility is enabled (not --no-repair)"],"tags":["compaction","index","issue347","thanos"],"backgroundTag":"schema-validation-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}