thanos-io/thanos · warning

remote query error ( )

Error message

remote query error (%s): %s

What it means

When a remote (federated) Prometheus query fails, the remote query engine wraps the failure. If partial responses are enabled, the error is downgraded to a warning 'remote query error (<remoteAddr>): <err>' attached to the result annotations instead of failing the whole query; otherwise the original error is returned as the result's Err.

Solutions

1. Inspect the inner error text after the address prefix; fix the underlying remote-side failure (connectivity, timeout, store error).
2. If complete results are required, disable partial response so the query fails fast with the original error.
3. Check the remote address r.remoteAddr for staleness (retired peers, wrong --endpoint flags).
4. Retry the query; transient remote failures often resolve once the remote querier/store recovers.

Example fix

// before (PartialResponse=true hides the failure as a warning)
engine.NewRemoteQuery(..., opts.WithPartialResponse(true))
// after: surface the real error
engine.NewRemoteQuery(..., opts.WithPartialResponse(false))

// pre-check: decide policy before issuing query
if requireCompleteResults && opts.PartialResponse {
    return fmt.Errorf("partial response enabled; remote errors will be downgraded to warnings")
}

res := qry.Exec(ctx)
if res.Err != nil {
    // partial response disabled: real error
    return fmt.Errorf("remote query failed: %w", res.Err)
}
for _, w := range res.Warnings {
    if strings.Contains(w.Error(), "remote query error") {
        // partial data path: decide to fail or proceed
    }
}

When it happens

Trigger: remoteQuery.responseForError is called with a non-nil error received while streaming a Query/QueryRange response from the remote Thanos endpoint, and r.opts.PartialResponse is true.

Common situations: Remote storegw/querier returns an error mid-stream (store unreachable, query timeout, upstream query failed) while the client configured --query.partial-response to tolerate partial data.

Related errors

• remote query warning
• sync before first pass of downsampling
• sync before second pass of downsampling
• error starting web server
• no query API server reachable

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4b1d342ad356597d. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query/remote_engine.go:295

	logger log.Logger
	client Client
	opts   Opts

	plan       api.RemoteQuery
	start      time.Time
	end        time.Time
	interval   time.Duration
	remoteAddr string

	samplesStats *stats.QuerySamples

	cancel context.CancelFunc
}

func (r *remoteQuery) responseForError(err error) *promql.Result {
	if r.opts.PartialResponse {
		return &promql.Result{
			Warnings: annotations.New().Add(fmt.Errorf("remote query error (%s): %s", r.remoteAddr, err)),
		}
	}
	return &promql.Result{Err: err}
}

func (r *remoteQuery) Exec(ctx context.Context) *promql.Result {
	start := time.Now()
	defer func() {
		keys := []any{
			"msg", "Executed remote query",
			"query", r.plan.String(),
			"time", time.Since(start),
		}
		if r.samplesStats != nil {
			keys = append(keys, "remote_peak_samples", r.samplesStats.PeakSamples)
			keys = append(keys, "remote_total_samples", r.samplesStats.TotalSamples)
		}
		level.Debug(r.logger).Log(keys...)

View on GitHub (pinned to 35b8b99117)