{"record":{"id":"7bf2cb452ec41de7","repo":"hashicorp/nomad","slug":"server-setup-failed-v","errorCode":null,"errorMessage":"server setup failed: %v","messagePattern":"server setup failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"command/agent/agent.go","lineNumber":1204,"sourceCode":"\n\t// Generate a node ID and persist it if it is the first instance, otherwise\n\t// read the persisted node ID.\n\tif err := a.setupNodeID(conf); err != nil {\n\t\treturn fmt.Errorf(\"setting up server node ID failed: %s\", err)\n\t}\n\n\t// Sets up the keyring for gossip encryption\n\tif err := a.setupKeyrings(conf); err != nil {\n\t\treturn fmt.Errorf(\"failed to configure keyring: %v\", err)\n\t}\n\n\t// Create the server\n\tserver, err := nomad.NewServer(conf,\n\t\ta.consulCatalog,           // self service discovery\n\t\ta.consulConfigEntriesFunc, // writing config entries for gateways\n\t)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"server setup failed: %v\", err)\n\t}\n\ta.server = server\n\n\t// Consul check addresses default to bind but can be toggled to use advertise\n\trpcCheckAddr := a.config.normalizedAddrs.RPC\n\tserfCheckAddr := a.config.normalizedAddrs.Serf\n\n\tdefaultConsul := conf.ConsulConfigs[structs.ConsulDefaultCluster]\n\n\tif *defaultConsul.ChecksUseAdvertise {\n\t\trpcCheckAddr = a.config.AdvertiseAddrs.RPC\n\t\tserfCheckAddr = a.config.AdvertiseAddrs.Serf\n\t}\n\n\t// Create the Nomad Server services for Consul\n\tif *defaultConsul.AutoAdvertise {\n\t\thttpServ := &structs.Service{\n\t\t\tName:      defaultConsul.ServerServiceName,","sourceCodeStart":1186,"sourceCodeEnd":1222,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/agent/agent.go#L1186-L1222","documentation":"setupServer constructs the actual Nomad server via nomad.NewServer; any error from server construction (Raft setup, Serf start, state store open, TLS, etc.) is wrapped as 'server setup failed' and NewAgent aborts.","triggerScenarios":"nomad.NewServer returns an error during agent startup: cannot bind Raft/Serf ports, corrupted raft/raft.peers or state store in data_dir, invalid TLS material at runtime, or invalid runtime-tunable config values.","commonSituations":"Port conflicts on 4647/serf ports, corrupted data_dir after unclean shutdown or version downgrade, mismatched TLS certs/keys, stale raft state after changing node ID, or invalid autopilot/raft protocol settings.","solutions":["Read the wrapped %v detail for the underlying cause","Check for port conflicts (lsof/ss on 4647, 4648) and change ports or stop the conflicting process","Back up then inspect/repair data_dir/server state; never run a newer data_dir with an older Nomad binary","Verify TLS certificate/key/CA validity and hostname match if TLS is enabled","Ensure node ID matches prior state, or restore a consistent data_dir backup"],"exampleFix":"// before (shell)\nnomad agent -server ...  # Error: server setup failed: ...bind: address already in use\n// after (shell)\nss -lntp | grep 4647  # stop the conflicting process or set ports { http=4646 rpc=4647 serf=4648 }\nrm -f stale raft state only with a verified backup","handlingStrategy":"try-catch","validationCode":"// Pre-flight: check Raft/Serf ports are free\nfor _, p := range []string{\":4647\", \":4648\"} {\n    if ln, err := net.Listen(\"tcp\", p); err != nil {\n        log.Fatalf(\"port %s in use: %v\", p, err)\n    } else {\n        ln.Close()\n    }\n}","typeGuard":null,"tryCatchPattern":"server, err := nomad.NewServer(conf, consulCatalog, consulConfigEntriesFunc)\nif err != nil {\n    if strings.Contains(err.Error(), \"address already in use\") {\n        return fmt.Errorf(\"server setup failed (port conflict): %w\", err)\n    }\n    return fmt.Errorf(\"server setup failed: %w\", err)\n}","preventionTips":["Never run a Nomad binary older than the one that wrote data_dir","Monitor and reserve ports 4646/4647/4648; check with ss/lsof before start","Back up data_dir before upgrades and unclean shutdowns","Validate TLS material (expiry, hostname SANs) before enabling TLS","Keep node_id and data_dir consistent across restarts"],"tags":["nomad","server","startup","raft","serf"],"backgroundTag":"server-setup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}