thanos-io/thanos · error

proxy Series()

Error message

proxy Series()

What it means

selectFn issues a gRPC Series() call through the store proxy (fan-out to store/querier nodes). Any transport, RPC, or remote-store error returned by proxy.Series is wrapped as 'proxy Series()'. This is the umbrella error for store-gRPC failures during a series query.

Solutions

1. Check connectivity to the store endpoints (DNS, ports, TLS)
2. Inspect store-gateway/sidecar logs for the underlying RPC error
3. Increase --store.grpc.series-timeout / query timeout if queries are large
4. Verify store nodes are registered and healthy in the query's store API set

Example fix

// before: proxy.Series fails on dead store
// after: check/restart store and confirm with grpc probe
thanos store endpoints:
  - store-gateway.thanos:10901
# verify: grpcurl -plaintext store-gateway.thanos:10901 list

// probe stores before issuing queries
for _, s := range stores {
    if !grpcHealthy(ctx, s.Addr, 2*time.Second) {
        return fmt.Errorf("store %s unhealthy", s.Addr)
    }
}

if err := ss.Err(); err != nil {
    if strings.Contains(err.Error(), "proxy Series()") {
        // partial store failure: retry with backoff, or filter dead stores
        return retryBackoff(ctx, func() error { return runQuery() })
    }
    return err
}

When it happens

Trigger: Querier Select fans out to stores and one or more gRPC Series calls fail: connection refused, TLS issues, remote side returns an error, deadline exceeded.

Common situations: Store gateway down or unreachable, DNS/service discovery failures in Kubernetes, TLS certificate mismatch, per-store timeouts on large queries, network partition.

Related errors

• no query API server reachable
• proxy LabelValues()
• proxy LabelNames()
• no matchers specified (excluding selector labels)
• send series response

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/75f142db0dd769ff. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query/querier.go:400

		ShardInfo:               q.shardInfo,
		PartialResponseStrategy: q.partialResponseStrategy,
		SkipChunks:              q.skipChunks,
		ResponseBatchSize:       int64(q.seriesResponseBatchSize),
		QueryHints: &storepb.QueryHints{
			ProjectionLabels:  hints.ProjectionLabels,
			ProjectionInclude: hints.ProjectionInclude,
			// NOTE(GiedriusS): this is what the Thanos parquet gateway uses right now so
			// I just copied/pasted it.
			SeriesHashLabelName: SeriesHashLabelName,
		},
	}
	if q.isDedupEnabled() {
		// Soft ask to sort without replica labels and push them at the end of labelset.
		req.WithoutReplicaLabels = q.replicaLabels
	}

	if err := q.proxy.Series(&req, resp); err != nil {
		return nil, storepb.SeriesStatsCounter{}, errors.Wrap(err, "proxy Series()")
	}
	warns := annotations.New().Merge(resp.warnings)

	if !q.isDedupEnabled() {
		return NewPromSeriesSet(
			newStoreSeriesSet(resp.seriesSet),
			q.mint,
			q.maxt,
			aggrs,
			warns,
		), resp.seriesSetStats, nil
	}

	// TODO(bwplotka): Move to deduplication on chunk level inside promSeriesSet, similar to what we have in dedup.NewDedupChunkMerger().
	// This however require big refactor, caring about correct AggrChunk to iterator conversion and counter reset apply.
	// For now we apply simple logic that splits potential overlapping chunks into separate replica series, so we can split the work.
	set := NewPromSeriesSet(
		dedup.NewOverlapSplit(newStoreSeriesSet(resp.seriesSet)),

View on GitHub (pinned to 35b8b99117)