{"record":{"id":"0cfdca7cf8d5b323","repo":"hashicorp/nomad","slug":"failed-to-start-rpc-layer-v","errorCode":null,"errorMessage":"Failed to start RPC layer: %v","messagePattern":"Failed to start RPC layer: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/server.go","lineNumber":474,"sourceCode":"\t}\n\n\t// Set up the SSO OIDC provider cache. This is needed by the setupRPC, but\n\t// must be done separately so that the server can stop all background\n\t// processes when it shuts down itself.\n\ts.oidcProviderCache = oidc.NewProviderCache()\n\n\t// Set up OIDC requests cache for state that persists between calls to\n\t// ACL.OIDCAuthURL and ACL.OIDCCompleteAuth.\n\t// It needs no special handling to handle agent shutdowns (its Store method\n\t// handles this lifecycle).\n\t// 6 minutes is 1 minute longer than the JWT expiration time in the cap lib.\n\ts.oidcRequestCache = oidc.NewRequestCache(6 * time.Minute)\n\n\t// Initialize the RPC layer\n\tif err := s.setupRPC(tlsWrap); err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start RPC layer\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start RPC layer: %v\", err)\n\t}\n\n\ts.auth = auth.NewAuthenticator(&auth.AuthenticatorConfig{\n\t\tStateFn:        s.State,\n\t\tLogger:         s.logger,\n\t\tGetLeaderACLFn: s.getLeaderAcl,\n\t\tAclsEnabled:    s.config.ACLEnabled,\n\t\tVerifyTLS:      s.config.TLSConfig != nil && s.config.TLSConfig.EnableRPC && s.config.TLSConfig.VerifyServerHostname,\n\t\tRegion:         s.Region(),\n\t\tEncrypter:      s.encrypter,\n\t})\n\n\t// Initialize the Raft server\n\tif err := s.setupRaft(); err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start Raft\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start Raft: %v\", err)\n\t}","sourceCodeStart":456,"sourceCodeEnd":492,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L456-L492","documentation":"NewServer wraps any error from setupRPC (binding RPC listeners, registering handlers, TLS wrapping) with \"Failed to start RPC layer\" after shutting the partially-built server down. The underlying cause is appended via %v, so the wrapped message is what actually needs fixing.","triggerScenarios":"Starting a server where setupRPC fails — typically because the configured HTTP/RPC address or port is already in use, is unbindable, or TLS certificate/key material cannot be loaded to build the tlsWrap function.","commonSituations":"Port 4647 already bound by another Nomad instance or stale process; invalid TLS cert paths or expired certificates; binding to an IP not present on the host (e.g. advertise address mismatch in containers).","solutions":["Read the wrapped %v cause: if 'address already in use', free port 4647 or change bind_addr/ports.rpc","Verify TLSConfig cert/key/CA paths and validity, since TLS setup feeds setupRPC","Check bind_addr resolves to a local interface on the host","Run `ss -ltnp | grep 4647` (or lsof) to identify the conflicting process"],"exampleFix":"// before\nports { rpc = 4647 } // port already used by another agent\n// after\nports { rpc = 14647 } # or stop the conflicting process\nsudo systemctl stop nomad@legacy","handlingStrategy":"try-catch","validationCode":"// Pre-flight: check the RPC port is free and TLS material is readable\nif ln, err := net.Listen(\"tcp\", fmt.Sprintf(\"%s:%d\", cfg.BindAddr, cfg.Ports.RPC)); err != nil {\n    return fmt.Errorf(\"RPC port in use: %w\", err)\n} else { ln.Close() }\nfor _, p := range []string{cfg.TLSConfig.CertFile, cfg.TLSConfig.KeyFile} {\n    if _, err := os.Stat(p); err != nil { return fmt.Errorf(\"missing TLS file %s\", p) }\n}","typeGuard":null,"tryCatchPattern":"srv, err := nomad.NewServer(config, catalog, consulFn)\nif err != nil {\n    var rety net.Error\n    if strings.Contains(err.Error(), \"Failed to start RPC layer\") {\n        // inspect wrapped cause: port conflict vs TLS failure\n    }\n    return err\n}","preventionTips":["Check port 4647 availability before launching agents","Validate TLS cert/key paths and expiry as part of deploy preflight","Use unique bind/advertise addresses per agent when co-locating on one host"],"tags":["nomad","go","rpc","port-conflict","startup"],"backgroundTag":"port-already-in-use","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}