{"record":{"id":"9b5a84459362e805","repo":"thanos-io/thanos","slug":"send-request","errorCode":null,"errorMessage":"send request","messagePattern":"send request","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/store/prometheus.go","lineNumber":481,"sourceCode":"\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"marshal read request\")\n\t}\n\n\tu := *p.base\n\tu.Path = path.Join(u.Path, \"api/v1/read\")\n\n\tpreq, err := http.NewRequest(\"POST\", u.String(), bytes.NewReader(snappy.Encode(nil, reqb)))\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"unable to create request\")\n\t}\n\tpreq.Header.Add(\"Content-Encoding\", \"snappy\")\n\tpreq.Header.Set(\"Content-Type\", \"application/x-stream-protobuf\")\n\tpreq.Header.Set(\"X-Prometheus-Remote-Read-Version\", \"0.1.0\")\n\n\tpreq.Header.Set(\"User-Agent\", clientconfig.ThanosUserAgent)\n\tpresp, err = p.client.Do(preq.WithContext(ctx))\n\tif err != nil {\n\t\treturn nil, errors.Wrap(err, \"send request\")\n\t}\n\tif presp.StatusCode/100 != 2 {\n\t\t// Best effort read.\n\t\tb, err := io.ReadAll(presp.Body)\n\t\tif err != nil {\n\t\t\tlevel.Error(p.logger).Log(\"msg\", \"failed to read response from non 2XX remote read request\", \"err\", err)\n\t\t}\n\t\t_ = presp.Body.Close()\n\t\treturn nil, errors.Errorf(\"request failed with code %s; msg %s\", presp.Status, string(b))\n\t}\n\n\treturn presp, nil\n}\n\n// matchesExternalLabels returns false if given matchers are not matching external labels.\n// If true, matchesExternalLabels also returns Prometheus matchers without those matching external labels.\nfunc matchesExternalLabels(ms []storepb.LabelMatcher, externalLabels labels.Labels, cache storecache.MatchersCache) (bool, []*labels.Matcher, error) {\n\tvar (","sourceCodeStart":463,"sourceCodeEnd":499,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/store/prometheus.go#L463-L499","documentation":"startPromRemoteRead performs the POST via p.client.Do(preq.WithContext(ctx)); any transport-level failure (connection refused, DNS failure, timeout, context cancellation) is wrapped as \"send request\". It means the HTTP request to Prometheus's remote read endpoint never completed successfully.","triggerScenarios":"Prometheus unreachable at the configured address, TLS handshake failure, network partition, or the caller's context being canceled/timed out mid-request.","commonSituations":"Wrong host/port in store address; Prometheus not running or behind a firewall; mTLS/TLS cert issues; slow queries exceeding client timeout; query canceled by upstream (common in multi-tier Thanos setups).","solutions":["Confirm the Prometheus address is reachable: curl http://<addr>/api/v1/read (expect non-connection-error)","Check for context cancellation — increase caller/step timeouts if the query legitimately takes long","Verify TLS configuration (CA bundle, client certs) if using https","Check DNS/service discovery so the address resolves to a live instance"],"exampleFix":"// before: no per-query timeout control\ncctx := ctx\n// after: bound remote-read latency explicitly\nctx2, cancel := context.WithTimeout(ctx, 30*time.Second)\ndefer cancel()\npresp, err = p.client.Do(preq.WithContext(ctx2))","handlingStrategy":"retry","validationCode":"conn, err := net.DialTimeout(\"tcp\", host, 2*time.Second)\nif err != nil { return fmt.Errorf(\"prometheus unreachable: %w\", err) }\nconn.Close()","typeGuard":null,"tryCatchPattern":"var resp *storepb.SeriesResponse\nbackoff := 100 * time.Millisecond\nfor i := 0; i < 3; i++ {\n    resp, err = store.Series(ctx, req)\n    if err != nil && strings.Contains(err.Error(), \"send request\") && ctx.Err() == nil {\n        time.Sleep(backoff); backoff *= 2; continue\n    }\n    break\n}","preventionTips":["Add health checks against the Prometheus address before routing queries","Set generous but bounded client timeouts for remote-read","Check TLS certs and DNS in the deployment environment","Distinguish context-canceled errors (caller gave up) from transport failures"],"tags":["prometheus","network","http","timeout"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}