What causes "send request to %q" in thanos-io/thanos?

Check the wrapped cause and target URL in the message; verify the Alertmanager is reachable (curl the URL from the same host/network).

thanos-io/thanos · error

send request to

Error message

send request to %q

What it means

postAlerts sends alert batches over HTTP to the configured Alertmanager replicas. When the HTTP request itself fails (dial error, timeout, connection reset, TLS failure), the dispatcher error is wrapped with the target URL, producing 'send request to %q'. The underlying cause remains wrapped via pkg/errors so errors.Is/As still work.

Solutions

Check the wrapped cause and target URL in the message; verify the Alertmanager is reachable (curl the URL from the same host/network).
Confirm DNS, firewall, and TLS configuration for the alertmanager URLs.
Check whether the request was canceled by the send timeout and raise the timeout if targets are slow.
Ensure dispatcher/client settings (TLS, proxy, HTTP/2) match the Alertmanager deployment.

Example fix

// before
resp, err := a.dispatcher.Do(req)
if err != nil {
    return errors.Wrapf(err, "send request to %q", u.String())
}
// after — caller-side guard
if err := sender.postAlerts(ctx, alerts); err != nil {
    var netErr net.Error
    if errors.As(err, &netErr) && netErr.Timeout() {
        level.Warn(logger).Log("msg", "alert send timed out, will retry", "err", err)
    }
    return err
}

Defensive patterns

Strategy: retry

Validate before calling

u, err := url.Parse(amURL)
if err != nil || u.Scheme == "" || u.Host == "" {
    return fmt.Errorf("invalid alertmanager url %q", amURL)
}
// optionally: probe reachability with a HEAD/GET before alert push

Try / catch

var nerr net.Error
if errors.As(err, &nerr) && nerr.Timeout() {
    // retry with backoff
} else if isConnRefused(err) {
    // alertmanager down: alert on infra, backoff
}

Prevention

Health-check Alertmanager URLs before dispatch.
Set explicit send timeouts and retry with backoff for transient errors.
Keep dispatcher TLS/auth config in sync with target Alertmanager.
Monitor alert send failure rates and queue depth.

When it happens

Trigger: a.dispatcher.Do(req) returns an error: DNS failure, connection refused, request context canceled by the send timeout (defer cancel()), or TLS handshake failure while pushing alerts from the ruler/alertmanager to peer Alertmanagers.

Common situations: Alertmanager endpoints unreachable (wrong -alertmanager.url / cluster peers down); network partitions; firewalls blocking egress; alerts timing out because target Alertmanagers are overloaded.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/897e2997868f01e2. Report an issue: GitHub.

Appendix: source

Thrown at pkg/alert/alert.go:394

		dispatcher: dispatcher,
		timeout:    timeout,
		version:    version,
	}
}

func (a *Alertmanager) postAlerts(ctx context.Context, u url.URL, r io.Reader) error {
	req, err := http.NewRequest("POST", u.String(), r)
	if err != nil {
		return err
	}
	ctx, cancel := context.WithTimeout(ctx, a.timeout)
	defer cancel()
	req = req.WithContext(ctx)
	req.Header.Set("Content-Type", contentTypeJSON)

	resp, err := a.dispatcher.Do(req)
	if err != nil {
		return errors.Wrapf(err, "send request to %q", u.String())
	}
	defer runutil.ExhaustCloseWithLogOnErr(a.logger, resp.Body, "send one alert")

	if resp.StatusCode/100 != 2 {
		return errors.Errorf("bad response status %v from %q", resp.Status, u.String())
	}
	return nil
}

View on GitHub (pinned to 35b8b99117)