thanos-io/thanos · error

query Prometheus

Error message

query Prometheus

What it means

Series issues the remote-read HTTP request via p.startPromRemoteRead; if that returns an error, it is wrapped as 'query Prometheus'. This covers request construction, URL building, connection, and non-handled transport failures against the Prometheus remote read endpoint.

Solutions

1. Verify the Prometheus remote-read endpoint URL and that /api/v1/read responds (curl it from the store host).
2. Check network connectivity, DNS, firewalls, and service discovery between Thanos and Prometheus.
3. Confirm TLS settings (CA bundle, client certs) match the Prometheus server configuration.
4. Check the wrapped cause (%w) for timeout vs connection-refused details and adjust timeouts.

Example fix

// before
p, _ := NewPrometheusStore(..., "http://prometheus:9090/reading", ...)
// after
p, _ := NewPrometheusStore(..., "http://prometheus:9090/api/v1/read", ...)

u, err := url.Parse(remoteReadURL)
if err != nil || u.Scheme == "" || u.Host == "" {
    return fmt.Errorf("invalid remote read URL %q", remoteReadURL)
}
resp, err := http.Get(u.String() /* or HEAD on /api/v1/read */) // verify reachability first

if err := s.Series(ctx, req); err != nil {
    if strings.Contains(err.Error(), "query Prometheus") {
        // retry with backoff for transient network errors; check errors.Unwrap for connection-refused vs timeout
    }
    return err
}

When it happens

Trigger: Calling Series on a PrometheusStore whose endpoint is unreachable, has a bad URL/DNS name, is not serving /api/v1/read, TLS verification fails, or the request times out.

Common situations: Wrong Prometheus address in the store-gateway/proxy config; Prometheus not exposing remote read (no --enable-feature=remote-write-receiver variants or old version); network policies/firewalls; TLS/mTLS misconfiguration.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• copy response
• failed to validate prometheus flags
• perform request against
• request config against
• read query instant response

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/f6932e2728ec5599. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/prometheus.go:206

		case labels.MatchNotEqual:
			pm.Type = prompb.LabelMatcher_NEQ
		case labels.MatchRegexp:
			pm.Type = prompb.LabelMatcher_RE
		case labels.MatchNotRegexp:
			pm.Type = prompb.LabelMatcher_NRE
		default:
			return errors.New("unrecognized matcher type")
		}
		q.Matchers = append(q.Matchers, pm)
	}

	queryPrometheusSpan, ctx := tracing.StartSpan(s.Context(), "query_prometheus")
	queryPrometheusSpan.SetTag("query.request", q.String())

	httpResp, err := p.startPromRemoteRead(ctx, q)
	if err != nil {
		queryPrometheusSpan.Finish()
		return errors.Wrap(err, "query Prometheus")
	}

	// Negotiate content. We requested streamed chunked response type, but still we need to support old versions of
	// remote read.
	contentType := httpResp.Header.Get("Content-Type")
	if strings.HasPrefix(contentType, "application/x-protobuf") {
		return p.handleSampledPrometheusResponse(s, httpResp, queryPrometheusSpan, extLset, enableChunkHashCalculation, extLsetToRemove)
	}

	if !strings.HasPrefix(contentType, "application/x-streamed-protobuf; proto=prometheus.ChunkedReadResponse") {
		return errors.Errorf("not supported remote read content type: %s", contentType)
	}
	return p.handleStreamedPrometheusResponse(s, shardMatcher, httpResp, queryPrometheusSpan, extLset, enableChunkHashCalculation, extLsetToRemove)
}

func (p *PrometheusStore) handleSampledPrometheusResponse(
	s flushableServer,
	httpResp *http.Response,

View on GitHub (pinned to 35b8b99117)