thanos-io/thanos · error

failed to create HTTP query client

Error message

failed to create HTTP query client: %w

What it means

This error is produced when clientconfig.NewHTTPClient fails to construct an HTTP client from the http_client config of a query endpoint entry. runRule iterates queryCfg and returns this fmt.Errorf wrapping the underlying error. Typically it reflects invalid TLS/authorization settings that prevent building the http.RoundTripper.

Solutions

1. Verify every tls_cert/tls_key/ca_file path in the query config exists and is readable by the rule process.
2. Check bearer_token or bearer_token_file values are valid and the file is mounted.
3. Test the httpConfig with a minimal curl using the same certs to confirm they load.
4. Ensure Kubernetes Secrets holding TLS material are mounted into the rule pod at the configured paths.

Example fix

// before
http_config:
  tls_config:
    cert_file: /etc/thanos/tls/cert.pem  # not mounted
// after
http_config:
  tls_config:
    cert_file: /var/run/secrets/thanos-tls/tls.crt
    key_file: /var/run/secrets/thanos-tls/tls.key

for _, p := range []string{cfg.TLS.CertFile, cfg.TLS.KeyFile, cfg.TLS.CAFile, cfg.BearerTokenFile} {
    if p != "" {
        if _, err := os.ReadFile(p); err != nil {
            return fmt.Errorf("query http config file %q unreadable: %w", p, err)
        }
    }
}

When it happens

Trigger: For a queryCfg entry where cfg.HTTPConfig.NotEmpty(), NewHTTPClient(cfg.HTTPConfig.HTTPClientConfig, "query") errors — e.g. unreadable TLS cert/key files, bad bearer token file, or invalid HTTP client config fields.

Common situations: TLS certificate or key file paths in the query config YAML do not exist inside the container; ca_file missing; bearer_token_file not mounted; malformed TLS config produced by secrets rotation.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• getting http client config
• Improper http client config
• create bucket compactor
• building gRPC client
• unable to load config file

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7ebce7aee264b2db. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:402

		logger,
		extprom.WrapRegistererWithPrefix("thanos_rule_query_apis_", reg),
		dns.ResolverType(conf.query.dnsSDResolver),
	)
	var (
		queryClients    []*clientconfig.HTTPClient
		promClients     []*promclient.Client
		grpcEndpointSet *query.EndpointSet
		grpcEndpoints   []string
	)

	queryClientMetrics := extpromhttp.NewClientMetrics(extprom.WrapRegistererWith(prometheus.Labels{"client": "query"}, reg))

	for _, cfg := range queryCfg {
		if cfg.HTTPConfig.NotEmpty() {
			cfg.HTTPConfig.HTTPClientConfig.ClientMetrics = queryClientMetrics
			c, err := clientconfig.NewHTTPClient(cfg.HTTPConfig.HTTPClientConfig, "query")
			if err != nil {
				return fmt.Errorf("failed to create HTTP query client: %w", err)
			}
			c.Transport = tracing.HTTPTripperware(logger, c.Transport)
			queryClient, err := clientconfig.NewClient(logger, cfg.HTTPConfig.EndpointsConfig, c, queryProvider.Clone())
			if err != nil {
				return fmt.Errorf("failed to create query client: %w", err)
			}
			queryClients = append(queryClients, queryClient)
			promClients = append(promClients, promclient.NewClient(queryClient, logger, "thanos-rule"))
			// Discover and resolve query addresses.
			addDiscoveryGroups(g, queryClient, conf.query.dnsSDInterval, logger)
		}

		if cfg.GRPCConfig != nil {
			grpcEndpoints = append(grpcEndpoints, cfg.GRPCConfig.EndpointAddrs...)
		}
	}

	if len(grpcEndpoints) > 0 {

View on GitHub (pinned to 35b8b99117)