thanos-io/thanos · warning · issue347Error

invalid, but reparable block

Error message

invalid, but reparable block %s

What it means

The block is affected by the 'issue 347' bug (chunks referenced in the index fall outside the block's time range). Such a block is invalid but reparable: Thanos wraps it in a typed issue347Error with the ULID so the syncer can later repair the block's meta/index instead of failing compaction outright.

Solutions

1. Run 'thanos tools bucket verify --repair' to repair blocks affected by issue 347
2. Or delete the block and re-upload from source Prometheus if repair is not possible
3. Keep Thanos/Prometheus updated; blocks produced by fixed writers won't exhibit this

Example fix

// before
# thanos tools bucket verify --objstore.bucket prom // reports issue347 blocks
// after
# thanos tools bucket verify --repair --objstore.bucket prom

// Detect issue-347 blocks before compaction:
thanos tools bucket verify --objstore.bucket prom | grep -i issue347

When it happens

Trigger: stats.Issue347OutsideChunksErr() is non-nil during index health validation of a downloaded block — index references chunks outside [MinTime, MaxTime].

Common situations: Blocks produced by Prometheus versions affected by issue #347; old buckets that never ran the repair tool; migration of legacy data.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• gather index issues for block
• block with not healthy index found
• blocks with out-of-order chunks are dropped from compaction
• block id , try running with --debug.accept-malformed-index
• create bucket compactor

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/1586d21b6da7c3ca. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1242

				// Ensure all input blocks are valid.
				var stats block.HealthStats
				if err := tracing.DoInSpanWithErr(ctx, "compaction_block_health_stats", func(ctx context.Context) (e error) {
					stats, e = block.GatherIndexHealthStats(ctx, cg.logger, filepath.Join(bdir, block.IndexFilename), meta.MinTime, meta.MaxTime)
					return e
				}, opentracing.Tags{"block.id": meta.ULID}); err != nil {
					return errors.Wrapf(err, "gather index issues for block %s", bdir)
				}

				if err := stats.CriticalErr(); err != nil {
					return halt(errors.Wrapf(err, "block with not healthy index found %s; Compaction level %v; Labels: %v", bdir, meta.Compaction.Level, meta.Thanos.Labels))
				}

				if err := stats.OutOfOrderChunksErr(); err != nil {
					return outOfOrderChunkError(errors.Wrapf(err, "blocks with out-of-order chunks are dropped from compaction:  %s", bdir), meta.ULID)
				}

				if err := stats.Issue347OutsideChunksErr(); err != nil {
					return issue347Error(errors.Wrapf(err, "invalid, but reparable block %s", bdir), meta.ULID)
				}

				if err := stats.OutOfOrderLabelsErr(); !cg.acceptMalformedIndex && err != nil {
					return errors.Wrapf(err,
						"block id %s, try running with --debug.accept-malformed-index", meta.ULID)
				}
				level.Debug(cg.logger).Log("msg", "verified block", "block", meta.ULID.String(), "duration", time.Since(start), "duration_ms", time.Since(start).Milliseconds())
				return nil
			})
		}(errCtx, m)

		toCompactDirs = append(toCompactDirs, bdir)
	}
	sourceBlockStr := fmt.Sprintf("%v", toCompactDirs)

	if err := g.Wait(); err != nil {
		return false, nil, err
	}

View on GitHub (pinned to 35b8b99117)