thanos-io/thanos · error

verify

Error message

verify %s

What it means

Manager.Verify runs each registered Verifier's Verify method and, on failure, wraps the error with "verify <issueID>" so the operator knows which issue's verifier failed. It surfaces the underlying verifier error attributed to its issue ID.

Solutions

1. Read the wrapped cause to identify which underlying operation failed
2. Re-run verification; many object store errors are transient
3. Isolate the failing block ID from verifier logs and inspect/repair it specifically
4. Increase timeout or ensure stable bucket connectivity for large buckets

for _, v := range verifiers {
	if err := v.Verify(vCtx, matcher); err != nil {
		log.Printf("verifier %s failed: %v", v.IssueID(), err) // inspect wrapped cause
	}
}

When it happens

Trigger: Any registered Verifier returns an error from its Verify method (e.g. bucket fetch failures, corrupted index verification failures, internal errors) during thanos verify.

Common situations: Object store access problems during verification; blocks with corrupted index failing the index verifier; context cancellation/timeout during a long verify run.

Related errors

• create bucket compactor
• %v
• get meta file
• filter blocks marked for deletion
• joined health-check error messages (AnyErr)

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/003e47e51aaaf052. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/verify.go:146

	}
}

// Verify verifies matching blocks using registered list of Verifier and VerifierRepairer.
// TODO(blotka): Wrap bucket with WrapWithMetrics and print metrics after each issue (e.g how many blocks where touched).
func (m *Manager) Verify(ctx context.Context, idMatcher func(ulid.ULID) bool) error {
	if len(m.vs.Verifiers)+len(m.vs.VerifierRepairers) == 0 {
		return errors.New("nothing to verify. No verifiers and verifierRepairers registered")
	}

	logger := log.With(m.Logger, "verifiers", strings.Join(append(m.vs.VerifiersIDs(), m.vs.VerifierRepairersIDs()...), ","))
	level.Info(logger).Log("msg", "Starting verify task")

	for _, v := range m.vs.Verifiers {
		vCtx := m.Context
		vCtx.Logger = log.With(logger, "verifier", v.IssueID())
		vCtx.Context = ctx
		if err := v.Verify(vCtx, idMatcher); err != nil {
			return errors.Wrapf(err, "verify %s", v.IssueID())
		}
	}
	for _, vr := range m.vs.VerifierRepairers {
		vCtx := m.Context
		vCtx.Context = ctx
		vCtx.Logger = log.With(logger, "verifier", vr.IssueID())
		if err := vr.VerifyRepair(vCtx, idMatcher, false); err != nil {
			return errors.Wrapf(err, "verify %s", vr.IssueID())
		}
	}

	level.Info(logger).Log("msg", "verify task completed")
	return nil
}

// VerifyAndRepair verifies and repairs matching blocks using registered list of VerifierRepairer.
// TODO(blotka): Wrap bucket with WrapWithMetrics and print metrics after each issue (e.g how many blocks where touched).
func (m *Manager) VerifyAndRepair(ctx context.Context, idMatcher func(ulid.ULID) bool) error {

View on GitHub (pinned to 35b8b99117)