{"record":{"id":"cabe3fe4f10f5395","repo":"jaegertracing/jaeger","slug":"failed-to-discover-elasticsearch-nodes-sniffing","errorCode":null,"errorMessage":"failed to discover Elasticsearch nodes (sniffing): %w","messagePattern":"failed to discover Elasticsearch nodes \\(sniffing\\): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/elasticsearch/esclient/transport.go","lineNumber":110,"sourceCode":"\tpool, err := newPool(transportOpts...)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to build transport pool: %w\", err)\n\t}\n\trc := &rawClient{pool: pool, base: base}\n\tif opts.discoverNodes {\n\t\t// Node discovery (sniffing): query one seed node once at startup and add\n\t\t// the cluster's other nodes to the pool. This is a one-shot call — no\n\t\t// background goroutine is scheduled (that would require a discovery\n\t\t// interval), so close() still has nothing to stop. Left off by default\n\t\t// because a cluster that publishes addresses the client cannot reach (a\n\t\t// common AWS/proxy setup) would break the pool. DiscoverNodesContext\n\t\t// tolerates a nil ctx (it falls back to context.Background()).\n\t\tif err := pool.DiscoverNodesContext(ctx); err != nil {\n\t\t\t// The discovery request opened a connection through base; release it,\n\t\t\t// since we are abandoning this client instead of returning it for the\n\t\t\t// caller to Close.\n\t\t\trc.close()\n\t\t\treturn nil, fmt.Errorf(\"failed to discover Elasticsearch nodes (sniffing): %w\", err)\n\t\t}\n\t}\n\treturn rc, nil\n}\n\n// perform sends req through the pool. req carries a relative path (e.g.\n// \"/_cluster/health\"); the pool selects a node and fills in its scheme and host.\nfunc (r *rawClient) perform(req *http.Request) (*http.Response, error) {\n\treturn r.pool.Perform(req)\n}\n","sourceCodeStart":92,"sourceCodeEnd":121,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/elasticsearch/esclient/transport.go#L92-L121","documentation":"When node discovery (sniffing) is enabled, newRawClient issues a one-shot DiscoverNodesContext call against a seed node at startup. If that request fails, the partially built client is closed (releasing idle connections on the base transport) and the failure is wrapped as \"failed to discover Elasticsearch nodes (sniffing)\". This means the client could not obtain the cluster's node list, so it aborts rather than run with only seed nodes.","triggerScenarios":"Calling NewClient/newRawClient with discoverNodes=true while the seed node is unreachable, returns an error/5xx, requires auth the base RoundTripper does not supply, or the ctx passed in is already cancelled/expired.","commonSituations":"Elasticsearch down or wrong port at startup; firewall/network policies blocking the seed node; sniffing enabled against AWS/proxied clusters that publish internal addresses the client cannot reach (the code comment notes this is why sniffing is off by default); Kubernetes DNS not yet resolvable when the Jaeger process starts.","solutions":["Check the wrapped inner error for the concrete HTTP/network failure and confirm the seed node is reachable from the client host (curl http://host:9200).","Verify auth/TLS settings on the base RoundTripper match the cluster (401/403 from sniffing means credentials are wrong).","Disable node discovery (sniffing) if the cluster sits behind a proxy or managed service that publishes unreachable publish_host addresses.","If the error is a context deadline, increase the startup timeout or ensure dependencies are ready before Jaeger starts.","Restart/retry once the cluster is up — this is a startup-time one-shot call, not retried automatically."],"exampleFix":"// before (config)\ndiscover_nodes: true   # fails behind AWS proxy\n// after\ndiscover_nodes: false  # rely on the configured seed URLs only","handlingStrategy":"try-catch","validationCode":"// before enabling sniffing, verify a seed node answers\nresp, err := http.Get(\"http://es-seed:9200/_cluster/health\")\nif err != nil {\n\treturn fmt.Errorf(\"seed node unreachable, sniffing will fail: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"client, err := esclient.NewClient(ctx, trt, opts)\nif err != nil && strings.Contains(err.Error(), \"sniffing\") {\n\t// decide: disable discovery and retry with seeds only, or surface as fatal\n\topts.DiscoverNodes = false\n\tclient, err = esclient.NewClient(ctx, trt, opts)\n}","preventionTips":["Leave node discovery off for managed clusters (AWS OpenSearch, proxies) that publish internal-only publish_host addresses.","Ensure the seed node is reachable and authenticated before startup (health-check with curl).","Set a generous startup context timeout so slow DNS/cluster startup does not cancel the sniffing call.","Run a readiness probe that depends on Elasticsearch availability before starting Jaeger."],"tags":["network","elasticsearch","sniffing","startup"],"backgroundTag":"node-discovery-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}