{"record":{"id":"35e60ff2a60c41a4","repo":"thanos-io/thanos","slug":"request-failed-with-code-s-msg-s","errorCode":null,"errorMessage":"request failed with code %s; msg %s","messagePattern":"request failed with code (.+?); msg (.+?)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/prometheus.go","lineNumber":490,"sourceCode":"\t\treturn nil, errors.Wrap(err, \"unable to create request\")\n\t}\n\tpreq.Header.Add(\"Content-Encoding\", \"snappy\")\n\tpreq.Header.Set(\"Content-Type\", \"application/x-stream-protobuf\")\n\tpreq.Header.Set(\"X-Prometheus-Remote-Read-Version\", \"0.1.0\")\n\n\tpreq.Header.Set(\"User-Agent\", clientconfig.ThanosUserAgent)\n\tpresp, err = p.client.Do(preq.WithContext(ctx))\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"send request\")\n\t}\n\tif presp.StatusCode/100 != 2 {\n\t\t// Best effort read.\n\t\tb, err := io.ReadAll(presp.Body)\n\t\tif err != nil {\n\t\t\tlevel.Error(p.logger).Log(\"msg\", \"failed to read response from non 2XX remote read request\", \"err\", err)\n\t\t}\n\t\t_ = presp.Body.Close()\n\t\treturn nil, errors.Errorf(\"request failed with code %s; msg %s\", presp.Status, string(b))\n\t}\n\n\treturn presp, nil\n}\n\n// matchesExternalLabels returns false if given matchers are not matching external labels.\n// If true, matchesExternalLabels also returns Prometheus matchers without those matching external labels.\nfunc matchesExternalLabels(ms []storepb.LabelMatcher, externalLabels labels.Labels, cache storecache.MatchersCache) (bool, []*labels.Matcher, error) {\n\tvar (\n\t\ttms []*labels.Matcher\n\t\terr error\n\t)\n\n\ttms, err = storecache.MatchersToPromMatchersCached(cache, ms...)\n\tif err != nil {\n\t\treturn false, nil, err\n\t}\n","sourceCodeStart":472,"sourceCodeEnd":508,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/prometheus.go#L472-L508","documentation":"When the remote read endpoint returns a non-2xx status, startPromRemoteRead reads the body (best effort) and returns errors.Errorf(\"request failed with code %s; msg %s\", presp.Status, string(b)). The error carries the HTTP status and whatever error text the server returned.","triggerScenarios":"Prometheus returns 4xx/5xx for /api/v1/read — e.g. 400 for an invalid query, 403 from an auth proxy, 422/500 for unsupported response types (streamed chunks requested of an old Prometheus), or 502 from a broken proxy.","commonSituations":"Requesting AcceptResponseTypes STREAMED_XOR_CHUNKS against a Prometheus < 2.13 that rejects it; auth proxy (oauth2_proxy/ForwardAuth) rejecting the request; query outside retention returning 4xx; rate limiting 429 at the proxy.","solutions":["Read the embedded msg in the error — it contains the server's explanation of the non-2xx status","If the status mentions unsupported response types, restrict remoteReadAcceptableResponses to non-streamed (or upgrade Prometheus to >= 2.13)","Fix authentication at any reverse proxy in front of Prometheus (headers, client certs, tokens)","Check Prometheus server logs at the same timestamp for the corresponding server-side error"],"exampleFix":"// before: request streamed chunks from an old Prometheus\nremoteReadAcceptableResponses = []prompb.ReadRequest_ResponseType{\n    prompb.ReadRequest_STREAMED_XOR_CHUNKS,\n}\n// after: fall back to sampled responses\nremoteReadAcceptableResponses = []prompb.ReadRequest_ResponseType{\n    prompb.ReadRequest_SAMPLES,\n}","handlingStrategy":"fallback","validationCode":"healthResp, err := http.Get(baseURL + \"/-/healthy\")\nif err != nil || healthResp.StatusCode != 200 { return fmt.Errorf(\"prometheus unhealthy: status %d\", statusCode(healthResp)) }","typeGuard":null,"tryCatchPattern":"resp, err := store.Series(ctx, req)\nif err != nil && strings.Contains(err.Error(), \"request failed with code\") {\n    // parse embedded status; if it indicates unsupported streamed chunks, retry with SAMPLES-only response type\n}","preventionTips":["Match remoteReadAcceptableResponses to the server's Prometheus version","Keep auth proxy configuration in sync with remote-read headers","Alert on non-2xx rates for /api/v1/read","Check server logs for the server-side cause paired with each status code"],"tags":["prometheus","http","remote-read","http-error"],"backgroundTag":"http-error-response","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}