{"record":{"id":"4b1d342ad356597d","repo":"thanos-io/thanos","slug":"remote-query-error-s-s","errorCode":null,"errorMessage":"remote query error (%s): %s","messagePattern":"remote query error \\((.+?)\\): (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/query/remote_engine.go","lineNumber":295,"sourceCode":"\tlogger log.Logger\n\tclient Client\n\topts   Opts\n\n\tplan       api.RemoteQuery\n\tstart      time.Time\n\tend        time.Time\n\tinterval   time.Duration\n\tremoteAddr string\n\n\tsamplesStats *stats.QuerySamples\n\n\tcancel context.CancelFunc\n}\n\nfunc (r *remoteQuery) responseForError(err error) *promql.Result {\n\tif r.opts.PartialResponse {\n\t\treturn &promql.Result{\n\t\t\tWarnings: annotations.New().Add(fmt.Errorf(\"remote query error (%s): %s\", r.remoteAddr, err)),\n\t\t}\n\t}\n\treturn &promql.Result{Err: err}\n}\n\nfunc (r *remoteQuery) Exec(ctx context.Context) *promql.Result {\n\tstart := time.Now()\n\tdefer func() {\n\t\tkeys := []any{\n\t\t\t\"msg\", \"Executed remote query\",\n\t\t\t\"query\", r.plan.String(),\n\t\t\t\"time\", time.Since(start),\n\t\t}\n\t\tif r.samplesStats != nil {\n\t\t\tkeys = append(keys, \"remote_peak_samples\", r.samplesStats.PeakSamples)\n\t\t\tkeys = append(keys, \"remote_total_samples\", r.samplesStats.TotalSamples)\n\t\t}\n\t\tlevel.Debug(r.logger).Log(keys...)","sourceCodeStart":277,"sourceCodeEnd":313,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/query/remote_engine.go#L277-L313","documentation":"When a remote (federated) Prometheus query fails, the remote query engine wraps the failure. If partial responses are enabled, the error is downgraded to a warning 'remote query error (<remoteAddr>): <err>' attached to the result annotations instead of failing the whole query; otherwise the original error is returned as the result's Err.","triggerScenarios":"remoteQuery.responseForError is called with a non-nil error received while streaming a Query/QueryRange response from the remote Thanos endpoint, and r.opts.PartialResponse is true.","commonSituations":"Remote storegw/querier returns an error mid-stream (store unreachable, query timeout, upstream query failed) while the client configured --query.partial-response to tolerate partial data.","solutions":["Inspect the inner error text after the address prefix; fix the underlying remote-side failure (connectivity, timeout, store error).","If complete results are required, disable partial response so the query fails fast with the original error.","Check the remote address r.remoteAddr for staleness (retired peers, wrong --endpoint flags).","Retry the query; transient remote failures often resolve once the remote querier/store recovers."],"exampleFix":"// before (PartialResponse=true hides the failure as a warning)\nengine.NewRemoteQuery(..., opts.WithPartialResponse(true))\n// after: surface the real error\nengine.NewRemoteQuery(..., opts.WithPartialResponse(false))","handlingStrategy":"try-catch","validationCode":"// pre-check: decide policy before issuing query\nif requireCompleteResults && opts.PartialResponse {\n    return fmt.Errorf(\"partial response enabled; remote errors will be downgraded to warnings\")\n}","typeGuard":null,"tryCatchPattern":"res := qry.Exec(ctx)\nif res.Err != nil {\n    // partial response disabled: real error\n    return fmt.Errorf(\"remote query failed: %w\", res.Err)\n}\nfor _, w := range res.Warnings {\n    if strings.Contains(w.Error(), \"remote query error\") {\n        // partial data path: decide to fail or proceed\n    }\n}","preventionTips":["Only enable partial response when incomplete results are acceptable.","Monitor result warnings for 'remote query error' strings in dashboards.","Keep remote endpoint discovery (DNS/SRV) healthy so addresses don't go stale.","Set sane remote-side timeouts to fail fast instead of hanging."],"tags":["remote-query","promql","partial-response","network"],"backgroundTag":"upstream-api-error","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}