thanos-io/thanos · error

verify and repair

Error message

verify and repair %s

What it means

One registered VerifierRepairer's VerifyRepair step failed while verifying-and-repairing its issue class; the wrapper adds the issue ID so the operator knows which verifier (e.g. index_replace, dedup) failed. The underlying cause ranges from bucket I/O errors to unrecoverable block corruption.

Solutions

1. Inspect the wrapped error for the underlying bucket or block failure
2. Check object-store access, permissions and connectivity
3. Re-run the verifier after fixing the root cause; repairers are idempotent
4. If a block is beyond repair, remove or re-upload it from its source

When it happens

Trigger: Thrown at pkg/verifier/verify.go:178 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/93c39dd16b1e3862. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/verify.go:178

}

// VerifyAndRepair verifies and repairs matching blocks using registered list of VerifierRepairer.
// TODO(blotka): Wrap bucket with WrapWithMetrics and print metrics after each issue (e.g how many blocks where touched).
func (m *Manager) VerifyAndRepair(ctx context.Context, idMatcher func(ulid.ULID) bool) error {
	if len(m.vs.Verifiers)+len(m.vs.VerifierRepairers) == 0 {
		return errors.New("nothing to verify. No verifierRepairers registered")
	}

	logger := log.With(m.Logger, "verifiers", strings.Join(m.vs.VerifierRepairersIDs(), ","))
	level.Warn(logger).Log("msg", "GLOBAL COMPACTOR SHOULD __NOT__ BE RUNNING ON THE SAME BUCKET")
	level.Info(logger).Log("msg", "Starting verify and repair task")

	for _, vr := range m.vs.VerifierRepairers {
		vCtx := m.Context
		vCtx.Logger = log.With(logger, "verifier", vr.IssueID())
		vCtx.Context = ctx
		if err := vr.VerifyRepair(vCtx, idMatcher, true); err != nil {
			return errors.Wrapf(err, "verify and repair %s", vr.IssueID())
		}
	}

	level.Info(logger).Log("msg", "verify and repair task completed")
	return nil
}

View on GitHub (pinned to 35b8b99117)