thanos-io/thanos · error · ApiError

retrieving exemplars

Error message

retrieving exemplars

What it means

Returned by the /api/v1/query_exemplars endpoint when client.Exemplars() — the RPC retrieving exemplar data from the upstream store — fails. Wrapped with errors.Wrap into an ErrorInternal (HTTP 500). It signals a backend retrieval failure, not a client input problem (bad start/end times fail earlier with BadData).

Solutions

1. Inspect the wrapped err for the transport-level cause.
2. Narrow the start/end time range to reduce load, and retry on timeouts.
3. Verify store-gateway/query nodes serving exemplars are healthy and registered.
4. Check frontend-to-store TLS and gRPC connectivity.

if (+endTs - +startTs > 24*3600*1000) console.warn('large exemplar range; expect slow queries');

try { const r = await fetch(exemplarsUrl); const b = await r.json(); if (b.errorType === 'internal') await backoffRetry(); } catch (e) { /* handle network failure */ }

When it happens

Trigger: Any GET /api/v1/query_exemplars where the Exemplars gRPC call errors: unreachable store/query node, canceled request context, deadline exceeded, or exemplar store RPC failure.

Common situations: Exemplar storage not backed by a healthy store-gateway, query node down, timeouts on large exemplar time ranges, TLS/network issues between frontend and stores.

Related errors

• retrieving targets
• error retrieving rules
• no query API server reachable
• failed to get tsdb status from prometheus
• proxy MetricMetadata

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/38cd31386e69e637. Report an issue: GitHub.

Appendix: source

Thrown at pkg/api/query/v1.go:1542

		}
		end, err := parseTimeParam(r, "end", v1.MaxTime)
		if err != nil {
			return nil, nil, &api.ApiError{Typ: api.ErrorBadData, Err: err}, func() {}
		}

		req := &exemplarspb.ExemplarsRequest{
			Start:                   timestamp.FromTime(start),
			End:                     timestamp.FromTime(end),
			Query:                   r.FormValue("query"),
			PartialResponseStrategy: ps,
		}

		tracing.DoInSpan(ctx, "retrieve_exemplars", func(ctx context.Context) {
			data, warnings, err = client.Exemplars(ctx, req)
		})

		if err != nil {
			return nil, nil, &api.ApiError{Typ: api.ErrorInternal, Err: errors.Wrap(err, "retrieving exemplars")}, func() {}
		}
		return data, warnings.AsErrors(), nil, func() {}
	}
}

func parseMetadataTimeRange(r *http.Request, defaultMetadataTimeRange time.Duration) (time.Time, time.Time, error) {
	// If start and end time not specified as query parameter, we get the range from the beginning of time by default.
	var defaultStartTime, defaultEndTime time.Time
	if defaultMetadataTimeRange == 0 {
		defaultStartTime = v1.MinTime
		defaultEndTime = v1.MaxTime
	} else {
		now := time.Now()
		defaultStartTime = now.Add(-defaultMetadataTimeRange)
		defaultEndTime = now
	}

	start, err := parseTimeParam(r, "start", defaultStartTime)

View on GitHub (pinned to 35b8b99117)