thanos-io/thanos · warning

remote query warning

Error message

remote query warning (%s): %s

What it means

While streaming a remote instant-query response, the remote endpoint sent a warnings field in a response message. The client records it as 'remote query warning (<remoteAddr>): <warn>' and continues consuming the stream; the query result carries these warnings as annotations, not an error.

Solutions

1. Read the warning text after the address prefix; address the remote-side cause if the results are unreliable.
2. Treat results as potentially incomplete; tighten fanout/limits on the remote querier to eliminate the warning.
3. If warnings must fail the query, disable partial response handling on the remote engine.
4. Check remote store/querier logs for the same warning to find the originating store.

Example fix

// warnings are annotations; inspect them after Exec
res := qry.Exec(ctx)
for _, w := range res.Warnings {
    log.Warnf("remote warning: %v", w)
}

func hasRemoteWarnings(res *promql.Result) bool {
    for _, w := range res.Warnings {
        if strings.Contains(w.Error(), "remote query warning") {
            return true
        }
    }
    return false
}

res := qry.Exec(ctx)
if len(res.Warnings) > 0 {
    log.Warnf("query completed with warnings: %v", res.Warnings)
}

When it happens

Trigger: During remoteQuery.Exec for an instant query, a streamed response message has non-empty msg.GetWarnings() and no transport error.

Common situations: Remote querier executed the query with warnings: too many samples, histogram/NaN caveats, clock-skew hints, or partial data notices from downstream stores.

Related errors

• remote query error ( )
• res.GetWarning() (propagated warning)
• at modifier after end
• negative offset
• Query not explainable

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cb9c05e55cf43caa. Report an issue: GitHub.

Appendix: source

Thrown at pkg/query/remote_engine.go:380

			return r.responseForError(err)
		}
		var (
			result   = make(promql.Vector, 0)
			warnings annotations.Annotations
			builder  = labels.NewScratchBuilder(8)
			qryStats querypb.QueryStats
		)
		for {
			msg, err := qry.Recv()
			if err == io.EOF {
				break
			}
			if err != nil {
				return r.responseForError(err)
			}

			if warn := msg.GetWarnings(); warn != "" {
				warnings.Add(errors.Errorf("remote query warning (%s): %s", r.remoteAddr, warn))
				continue
			}
			if s := msg.GetStats(); s != nil {
				qryStats = *s
				continue
			}
			if batch := msg.GetTimeseriesBatch(); batch != nil {
				for _, ts := range batch.Series {
					builder.Reset()
					for _, l := range ts.Labels {
						builder.Add(strings.Clone(l.Name), strings.Clone(l.Value))
					}
					if len(ts.Histograms) > 0 {
						result = append(result, promql.Sample{Metric: builder.Labels(), H: prompb.FromProtoHistogram(ts.Histograms[0]), T: r.start.UnixMilli()})
					} else {
						result = append(result, promql.Sample{Metric: builder.Labels(), F: ts.Samples[0].Value, T: r.start.UnixMilli()})
					}
				}

View on GitHub (pinned to 35b8b99117)