thanos-io/thanos · error

Aborted

Aborted

Error message

get exemplars for tenant %s

What it means

In the multi-TSDB exemplars path, Exemplars iterates every tenant's TSDB exemplars server and wraps any per-tenant failure with errors.Wrapf(err, "get exemplars for tenant %s", tenant), returned as a gRPC Aborted status. It tells you which tenant's store failed while serving exemplars.

Solutions

1. Identify the tenant named in the error message and inspect that tenant's TSDB logs for the wrapped cause.
2. Verify exemplars storage is enabled and healthy for that tenant (max_exemplars > 0).
3. Check the requested time range is within the tenant's retention/block layout.
4. Repair or reload the affected tenant's TSDB blocks if corruption is reported.

_, err := api.Exemplars(ctx, req)
if st, ok := status.FromError(err); ok && st.Code() == codes.Aborted {
	// message names the failing tenant; alert on that tenant's TSDB
	log.Errorf("tenant exemplars failed: %v", st.Message())
}

When it happens

Trigger: A multi-tenant ThanOS ruler/query setup where es.Exemplars(matchers, r.Start, r.End, s) fails for one tenant — e.g. corrupt head block, exemplars API disabled, or time-range beyond retention.

Common situations: Multi-tenant deployments where one tenant's TSDB has an issue while others serve fine; queries spanning tenants hit the first failing tenant and abort the whole request.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• %s
• empty exemplars data
• proxy Exemplars
• InvalidArgument
• Unknown

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b6856ab2f3dca1d3. Report an issue: GitHub.

Appendix: source

Thrown at pkg/exemplars/multitsdb.go:38

// NewMultiTSDB creates new exemplars.MultiTSDB.
func NewMultiTSDB(tsdbExemplarsServers func() map[string]*TSDB) *MultiTSDB {
	return &MultiTSDB{
		tsdbExemplarsServers: tsdbExemplarsServers,
	}
}

// Exemplars returns all specified exemplars from a MultiTSDB instance.
func (m *MultiTSDB) Exemplars(r *exemplarspb.ExemplarsRequest, s exemplarspb.Exemplars_ExemplarsServer) error {
	expr, err := extpromql.ParseExpr(r.Query)
	if err != nil {
		return status.Error(codes.Internal, err.Error())
	}
	matchers := parser.ExtractSelectors(expr)

	for tenant, es := range m.tsdbExemplarsServers() {
		if err := es.Exemplars(matchers, r.Start, r.End, s); err != nil {
			return status.Error(codes.Aborted, errors.Wrapf(err, "get exemplars for tenant %s", tenant).Error())
		}
	}
	return nil
}

View on GitHub (pinned to 35b8b99117)