thanos-io/thanos · error

failed to send series

Error message

failed to send series

What it means

limiterStore's Send wraps the seriesLimiter.Reserve error with "failed to send series" before forwarding a SeriesSet response. It means the number of series in the outgoing response would exceed the configured series limit, so the response is rejected instead of sent.

Solutions

1. Narrow the query matchers or time range to reduce selected series count.
2. Increase --store.series-limit on the relevant component.
3. Use partial-response / sharding to split the query across smaller requests.
4. Aggregate at the query-frontend to avoid unbounded fan-out.

Example fix

// before
query(query, matchers *[^.*])
// after
query(query, matchers limited to specific {job="prometheus", instance=~"specific-host.*"})

// pre-check cardinality via the query frontend before issuing Series
series, _ := promAPI.Series(ctx, matchers...)
if len(series) > seriesLimit { /* narrow matchers */ }

if err != nil && strings.Contains(err.Error(), "failed to send series") {
  // split the query (shard by label or time) and retry per-shard
}

When it happens

Trigger: Streaming a Series response where seriesCount added to previous reservations exceeds the series limiter's limit (Reserve fails, error wrapped by errors.Wrapf).

Common situations: Queries with very broad label selectors hitting --store.series-limit; dashboards issuing single queries over many series; limit set to a small value in thanos-query/store-gateway configuration.

Related errors

• limit violated (got )
• failed to send samples
• failed to parse template
• failed to execute template
• failed to parse the template

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/eb7d7232fb6efe4d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/limiter.go:187

func (i *limitedServer) Send(response *storepb.SeriesResponse) error {
	var seriesCount, chunksCount uint64
	if series := response.GetSeries(); series != nil {
		seriesCount = 1
		chunksCount = uint64(len(series.Chunks))
	} else if batch := response.GetBatch(); batch != nil {
		for _, series := range batch.Series {
			if series == nil {
				continue
			}
			seriesCount++
			chunksCount += uint64(len(series.Chunks))
		}
	} else {
		return i.Store_SeriesServer.Send(response)
	}

	if err := i.seriesLimiter.Reserve(seriesCount); err != nil {
		return errors.Wrapf(err, "failed to send series")
	}
	if err := i.samplesLimiter.Reserve(chunksCount * MaxSamplesPerChunk); err != nil {
		return errors.Wrapf(err, "failed to send samples")
	}

	return i.Store_SeriesServer.Send(response)
}

View on GitHub (pinned to 35b8b99117)