hashicorp/nomad · critical

Failed to start RPC layer: %v

Error message

Failed to start RPC layer: %v

What it means

NewServer wraps any error from setupRPC (binding RPC listeners, registering handlers, TLS wrapping) with "Failed to start RPC layer" after shutting the partially-built server down. The underlying cause is appended via %v, so the wrapped message is what actually needs fixing.

Source

Thrown at nomad/server.go:474

	}

	// Set up the SSO OIDC provider cache. This is needed by the setupRPC, but
	// must be done separately so that the server can stop all background
	// processes when it shuts down itself.
	s.oidcProviderCache = oidc.NewProviderCache()

	// Set up OIDC requests cache for state that persists between calls to
	// ACL.OIDCAuthURL and ACL.OIDCCompleteAuth.
	// It needs no special handling to handle agent shutdowns (its Store method
	// handles this lifecycle).
	// 6 minutes is 1 minute longer than the JWT expiration time in the cap lib.
	s.oidcRequestCache = oidc.NewRequestCache(6 * time.Minute)

	// Initialize the RPC layer
	if err := s.setupRPC(tlsWrap); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start RPC layer", "error", err)
		return nil, fmt.Errorf("Failed to start RPC layer: %v", err)
	}

	s.auth = auth.NewAuthenticator(&auth.AuthenticatorConfig{
		StateFn:        s.State,
		Logger:         s.logger,
		GetLeaderACLFn: s.getLeaderAcl,
		AclsEnabled:    s.config.ACLEnabled,
		VerifyTLS:      s.config.TLSConfig != nil && s.config.TLSConfig.EnableRPC && s.config.TLSConfig.VerifyServerHostname,
		Region:         s.Region(),
		Encrypter:      s.encrypter,
	})

	// Initialize the Raft server
	if err := s.setupRaft(); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start Raft", "error", err)
		return nil, fmt.Errorf("Failed to start Raft: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v cause: if 'address already in use', free port 4647 or change bind_addr/ports.rpc
2. Verify TLSConfig cert/key/CA paths and validity, since TLS setup feeds setupRPC
3. Check bind_addr resolves to a local interface on the host
4. Run `ss -ltnp | grep 4647` (or lsof) to identify the conflicting process

Example fix

// before
ports { rpc = 4647 } // port already used by another agent
// after
ports { rpc = 14647 } # or stop the conflicting process
sudo systemctl stop nomad@legacy

// Pre-flight: check the RPC port is free and TLS material is readable
if ln, err := net.Listen("tcp", fmt.Sprintf("%s:%d", cfg.BindAddr, cfg.Ports.RPC)); err != nil {
    return fmt.Errorf("RPC port in use: %w", err)
} else { ln.Close() }
for _, p := range []string{cfg.TLSConfig.CertFile, cfg.TLSConfig.KeyFile} {
    if _, err := os.Stat(p); err != nil { return fmt.Errorf("missing TLS file %s", p) }
}

srv, err := nomad.NewServer(config, catalog, consulFn)
if err != nil {
    var rety net.Error
    if strings.Contains(err.Error(), "Failed to start RPC layer") {
        // inspect wrapped cause: port conflict vs TLS failure
    }
    return err
}

When it happens

Trigger: Starting a server where setupRPC fails — typically because the configured HTTP/RPC address or port is already in use, is unbindable, or TLS certificate/key material cannot be loaded to build the tlsWrap function.

Common situations: Port 4647 already bound by another Nomad instance or stale process; invalid TLS cert paths or expired certificates; binding to an IP not present on the host (e.g. advertise address mismatch in containers).

Related errors

• failed to fetch signed identities: %w
• failed to fetch key from any peer: %v
• deprecated config field 'RaftBoltNoFreelistSync' is set; use
• Failed to start Raft: %v
• Failed to start serf: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0cfdca7cf8d5b323. Report an issue: GitHub.