jaegertracing/jaeger · error

failed to discover Elasticsearch nodes (sniffing): %w

Error message

failed to discover Elasticsearch nodes (sniffing): %w

What it means

When node discovery (sniffing) is enabled, newRawClient issues a one-shot DiscoverNodesContext call against a seed node at startup. If that request fails, the partially built client is closed (releasing idle connections on the base transport) and the failure is wrapped as "failed to discover Elasticsearch nodes (sniffing)". This means the client could not obtain the cluster's node list, so it aborts rather than run with only seed nodes.

Source

Thrown at internal/storage/elasticsearch/esclient/transport.go:110

	pool, err := newPool(transportOpts...)
	if err != nil {
		return nil, fmt.Errorf("failed to build transport pool: %w", err)
	}
	rc := &rawClient{pool: pool, base: base}
	if opts.discoverNodes {
		// Node discovery (sniffing): query one seed node once at startup and add
		// the cluster's other nodes to the pool. This is a one-shot call — no
		// background goroutine is scheduled (that would require a discovery
		// interval), so close() still has nothing to stop. Left off by default
		// because a cluster that publishes addresses the client cannot reach (a
		// common AWS/proxy setup) would break the pool. DiscoverNodesContext
		// tolerates a nil ctx (it falls back to context.Background()).
		if err := pool.DiscoverNodesContext(ctx); err != nil {
			// The discovery request opened a connection through base; release it,
			// since we are abandoning this client instead of returning it for the
			// caller to Close.
			rc.close()
			return nil, fmt.Errorf("failed to discover Elasticsearch nodes (sniffing): %w", err)
		}
	}
	return rc, nil
}

// perform sends req through the pool. req carries a relative path (e.g.
// "/_cluster/health"); the pool selects a node and fills in its scheme and host.
func (r *rawClient) perform(req *http.Request) (*http.Response, error) {
	return r.pool.Perform(req)
}

View on GitHub (pinned to 806f444784)

Solutions

1. Check the wrapped inner error for the concrete HTTP/network failure and confirm the seed node is reachable from the client host (curl http://host:9200).
2. Verify auth/TLS settings on the base RoundTripper match the cluster (401/403 from sniffing means credentials are wrong).
3. Disable node discovery (sniffing) if the cluster sits behind a proxy or managed service that publishes unreachable publish_host addresses.
4. If the error is a context deadline, increase the startup timeout or ensure dependencies are ready before Jaeger starts.
5. Restart/retry once the cluster is up — this is a startup-time one-shot call, not retried automatically.

Example fix

// before (config)
discover_nodes: true   # fails behind AWS proxy
// after
discover_nodes: false  # rely on the configured seed URLs only

// before enabling sniffing, verify a seed node answers
resp, err := http.Get("http://es-seed:9200/_cluster/health")
if err != nil {
	return fmt.Errorf("seed node unreachable, sniffing will fail: %w", err)
}

client, err := esclient.NewClient(ctx, trt, opts)
if err != nil && strings.Contains(err.Error(), "sniffing") {
	// decide: disable discovery and retry with seeds only, or surface as fatal
	opts.DiscoverNodes = false
	client, err = esclient.NewClient(ctx, trt, opts)
}

When it happens

Trigger: Calling NewClient/newRawClient with discoverNodes=true while the seed node is unreachable, returns an error/5xx, requires auth the base RoundTripper does not supply, or the ctx passed in is already cancelled/expired.

Common situations: Elasticsearch down or wrong port at startup; firewall/network policies blocking the seed node; sniffing enabled against AWS/proxied clusters that publish internal addresses the client cannot reach (the code comment notes this is why sniffing is off by default); Kubernetes DNS not yet resolvable when the Jaeger process starts.

Related errors

• error creating Elasticsearch client: %w
• failed to create Elasticsearch client: %w
• invalid HTTP server host:port: %w
• invalid gRPC server host:port: %w
• query server failed to initialize listener: %w

AI-assisted analysis of jaegertracing/jaeger@806f444784 (2026-09-01). Data as JSON: /api/errors/cabe3fe4f10f5395. Report an issue: GitHub.