thanos-io/thanos · error

failed to send samples

Error message

failed to send samples

What it means

limiterStore's Send wraps the samplesLimiter.Reserve error with "failed to send samples"; the estimated sample count is chunksCount * MaxSamplesPerChunk and exceeding the configured chunk/sample limit aborts the response. This protects downstream queriers from huge chunk loads.

Solutions

1. Shorten the query time range or lower data resolution/downsampling usage.
2. Increase --store.chunk-limit on store-gateway/query.
3. Split the query into smaller ranges (query sharding / time partitioning).
4. Verify downsampling is enabled so long ranges use downsampled blocks.

Example fix

// before
query_range:start=now-30d, step=15s
// after
query_range:start=now-7d, step=60s  # or rely on 5m downsampled blocks

// estimate samples: chunks ≈ range/step; ensure under limit
if uint64(estimatedChunks)*MaxSamplesPerChunk > samplesLimit { /* reduce range or raise limit */ }

if err != nil && strings.Contains(err.Error(), "failed to send samples") {
  // retry with shorter time range or coarser step
}

When it happens

Trigger: Streaming a Series response whose chunks, converted to estimated samples (chunks * MaxSamplesPerChunk), exceed the samples/chunks limiter limit.

Common situations: Long time ranges with high-resolution data in one query; --store.chunk-limit set too low; queries hitting many blocks at once.

Related errors

• iterate chunk while re-encoding
• populateWithDelChunkSeriesIterator: unexpected empty chunk…
• limit violated (got )
• failed to send series
• failed to parse template

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8806c50528a881dd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/limiter.go:190

		seriesCount = 1
		chunksCount = uint64(len(series.Chunks))
	} else if batch := response.GetBatch(); batch != nil {
		for _, series := range batch.Series {
			if series == nil {
				continue
			}
			seriesCount++
			chunksCount += uint64(len(series.Chunks))
		}
	} else {
		return i.Store_SeriesServer.Send(response)
	}

	if err := i.seriesLimiter.Reserve(seriesCount); err != nil {
		return errors.Wrapf(err, "failed to send series")
	}
	if err := i.samplesLimiter.Reserve(chunksCount * MaxSamplesPerChunk); err != nil {
		return errors.Wrapf(err, "failed to send samples")
	}

	return i.Store_SeriesServer.Send(response)
}

View on GitHub (pinned to 35b8b99117)