{"record":{"id":"97a6012153321f04","repo":"hashicorp/nomad","slug":"contacted-d-nomad-servers-v","errorCode":null,"errorMessage":"contacted %d Nomad Servers: %v","messagePattern":"contacted (.+?) Nomad Servers: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/server.go","lineNumber":1108,"sourceCode":"\n\t\tif len(nomadServerServices) == 0 {\n\t\t\tif len(mErr.Errors) > 0 {\n\t\t\t\tpeersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))\n\t\t\t\treturn mErr.ErrorOrNil()\n\t\t\t}\n\n\t\t\t// Log the error and return nil so future handlers\n\t\t\t// can attempt to register the `nomad` service.\n\t\t\tpollInterval := peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor)\n\t\t\ts.logger.Trace(\"no Nomad Servers advertising Nomad service in Consul datacenters\", \"service_name\", nomadServerServiceName, \"datacenters\", dcs, \"retry\", pollInterval)\n\t\t\tpeersTimeout.Reset(pollInterval)\n\t\t\treturn nil\n\t\t}\n\n\t\tnumServersContacted, err := s.Join(nomadServerServices)\n\t\tif err != nil {\n\t\t\tpeersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))\n\t\t\treturn fmt.Errorf(\"contacted %d Nomad Servers: %v\", numServersContacted, err)\n\t\t}\n\n\t\tpeersTimeout.Reset(maxStaleLeadership)\n\t\ts.logger.Info(\"successfully contacted Nomad servers\", \"num_servers\", numServersContacted)\n\n\t\treturn nil\n\t}\n\n\t// Hacky replacement for old ConsulSyncer Periodic Handler.\n\tgo func() {\n\t\tlastOk := true\n\t\tsync := time.NewTimer(0)\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-sync.C:\n\t\t\t\td := defaultConsulDiscoveryInterval\n\t\t\t\tif err := bootstrapFn(); err != nil {\n\t\t\t\t\t// Only log if it worked last time","sourceCodeStart":1090,"sourceCodeEnd":1126,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1090-L1126","documentation":"In the fallback bootstrap path, the server collects discovered Nomad servers from Consul and calls s.Join() to have Raft contact them. If Join fails (or contacts 0 usable servers), this error reports how many servers were contacted along with the underlying error, and the poll timer is reset to retry. It signals the server still cannot find/join a quorum via Consul.","triggerScenarios":"Quorum lost, Consul discovery returns nomad service entries, but s.Join fails — discovered servers are unreachable on the serf/RPC ports, TLS/gossip keys mismatch, firewall blocking 4647/4648, or all listed servers are themselves dead.","commonSituations":"Firewall/security-group rules blocking RPC between nodes; stale Consul entries pointing at decommissioned servers; raft_version/gossip encryption key mismatch after upgrade; network partition healing slowly.","solutions":["Verify connectivity to the contacted servers on the serf (4648) and RPC (4647) ports from this node.","Purge stale nomad service entries in Consul (dead members still registered).","Confirm gossip encryption key and TLS config match across servers.","Ensure at least a quorum of original servers is running, or use `nomad server force-leave` on dead members and restart."],"exampleFix":"// before\n# security group: only 4646 open\n// after\n# allow inbound tcp 4647 and both tcp/udp 4648 between nomad servers","handlingStrategy":"retry","validationCode":"// preflight: can this server reach serf/RPC ports on each discovered peer?\nfor _, peer := range discoveredServers {\n    if err := net.DialTimeout(\"tcp\", fmt.Sprintf(\"%s:%d\", peer.Address, serfPort), 2*time.Second); err != nil {\n        log.Printf(\"peer %s unreachable: %v\", peer.Name, err)\n    }\n}","typeGuard":null,"tryCatchPattern":"// Join is retried by the bootstrap poll; automation should wait and re-check\nwaitForQuorumPeers(\"/v1/status/peers\", expectedQuorum, 10*time.Minute)","preventionTips":["Open tcp/4647 and tcp+udp/4648 between all Nomad servers.","Run `nomad server force-leave` for permanently dead members.","Keep gossip encryption keys and TLS configs identical cluster-wide."],"tags":["raft","join","consul","network"],"backgroundTag":"server-join-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}