thanos-io/thanos · error

downstream URL returned an error

Error message

downstream URL %s returned an error: %d

What it means

The downstream readiness check received a non-success HTTP response from <downstreamURL>/-/ready; the downstream query service reports it is not healthy or the request failed, so the frontend will keep probing until ready.

Solutions

1. Wait until the downstream query instance finishes starting up
2. Inspect downstream logs for startup/health failures
3. Verify the downstream URL is correct
4. Check for proxy/load-balancer interference with /-/ready

When it happens

Trigger: Thrown at cmd/thanos/query_frontend.go:426 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/3bdb7b5fb825cda3. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/query_frontend.go:426

			doCheckDownstream := func() (rerr error) {
				timeoutCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
				defer cancel()

				readinessUrl := cfg.DownstreamURL + "/-/ready"
				req, err := http.NewRequestWithContext(timeoutCtx, http.MethodGet, readinessUrl, nil)
				if err != nil {
					return errors.Wrap(err, "creating request to downstream URL")
				}

				resp, err := downstreamRT.RoundTrip(req)
				if err != nil {
					return errors.Wrapf(err, "roundtripping to downstream URL %s", readinessUrl)
				}
				defer runutil.CloseWithErrCapture(&rerr, resp.Body, "downstream health check response body")

				if resp.StatusCode/100 == 4 || resp.StatusCode/100 == 5 {
					return errors.Errorf("downstream URL %s returned an error: %d", readinessUrl, resp.StatusCode)
				}

				return nil
			}
			for {
				if !firstRun {
					select {
					case <-ctx.Done():
						return nil
					case <-time.After(10 * time.Second):
					}
				}
				firstRun = false

				if err := doCheckDownstream(); err != nil {
					statusProber.NotReady(err)
				} else {
					statusProber.Ready()

View on GitHub (pinned to 35b8b99117)