thanos-io/thanos · error

request snapshot against

Error message

request snapshot against %s

What it means

Client.Snapshot executes the snapshot POST request via the HTTP client within the request context. This error wraps any transport-level failure while requesting the snapshot, including the target URL in the message. It is thrown before any response is read.

Solutions

1. Check connectivity to the Prometheus endpoint (curl the URL from the error)
2. Increase or remove the caller's context deadline — snapshots can be slow on large TSDBs
3. Verify DNS/firewall/TLS configuration between caller and Prometheus
4. Retry with backoff on transient network errors

Example fix

// before
ctx := context.Background()
dir, err := client.Snapshot(ctx, base, false)
// after
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
dir, err := client.Snapshot(ctx, base, false)

conn, err := net.DialTimeout("tcp", "prometheus:9090", 5*time.Second)
if err != nil { return fmt.Errorf("prometheus unreachable: %w", err) }
conn.Close()

var dir string
err := backoff.Retry(func() error {
    var e error
    dir, e = client.Snapshot(ctx, base, skipHead)
    return e
}, backoff.WithContext(backoff.NewExponentialBackOff(), ctx))

When it happens

Trigger: c.Do(req.WithContext(ctx)) fails: DNS resolution failure, connection refused/timeout, TLS handshake error, or ctx cancelled/expired mid-request.

Common situations: Prometheus host unreachable (wrong address/firewall); network partitions during backup jobs; context deadline exceeded because snapshot of large TSDB takes longer than the caller's timeout; TLS certificate issues.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• read query range response
• request metric against
• send request
• error starting web server
• failed to validate prometheus flags

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/d7309b51c122a736. Report an issue: GitHub.

Appendix: source

Thrown at pkg/promclient/promclient.go:358

	u := *base
	u.Path = path.Join(u.Path, "/api/v1/admin/tsdb/snapshot")

	req, err := http.NewRequest(
		http.MethodPost,
		u.String(),
		strings.NewReader(url.Values{"skip_head": []string{strconv.FormatBool(skipHead)}}.Encode()),
	)
	if err != nil {
		return "", errors.Wrap(err, "create request")
	}
	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

	span, ctx := tracing.StartSpan(ctx, "/prom_snapshot HTTP[client]")
	defer span.Finish()

	resp, err := c.Do(req.WithContext(ctx))
	if err != nil {
		return "", errors.Wrapf(err, "request snapshot against %s", u.String())
	}
	defer runutil.ExhaustCloseWithLogOnErr(c.logger, resp.Body, "query body")

	b, err := io.ReadAll(resp.Body)
	if err != nil {
		return "", errors.New("failed to read body")
	}

	if resp.StatusCode != 200 {
		return "", errors.Errorf("is 'web.enable-admin-api' flag enabled? got non-200 response code: %v, response: %v", resp.StatusCode, string(b))
	}

	var d struct {
		Data struct {
			Name string `json:"name"`
		} `json:"data"`
	}
	if err := json.Unmarshal(b, &d); err != nil {

View on GitHub (pinned to 35b8b99117)