temporalio/temporal · critical

failed to start service

Error message

failed to start service %v: %w

What it means

Wraps errors returned by the Start method of each internal Temporal service application (frontend, history, matching, worker, internal-frontend). startServices iterates services in init order and collects every failure into a multierr, so one bad service does not stop the others from attempting to start. The %w cause carries the service-specific error (ringpop membership join failure, gRPC listener bind error, etc.).

Solutions

  1. Read the wrapped cause and the service name in the message to identify which service and what sub-error failed
  2. Check that all service ports (e.g. 7233/7234/7235/7239 per config) are free: lsof/ss on the host or a stale container
  3. Verify membership config (broadcast host, host:port list) is reachable from every node — a common cause in Docker/Kubernetes
  4. Check the shared dependencies of the failing service (DB, Elasticsearch, dynamicconfig source) are up
  5. For multi-error output, note multierr appends all failures — fix the root shared dependency if many services failed

Example fix

// before: config serves frontend on default 7233 but an old process still holds it
// failed to start service frontend: listen tcp :7233: bind: address already in use

// after: stop the stale process, then start
// kill <stale-pid>
if err := srv.Start(ctx); err != nil {
	log.Fatalf("server failed: %v", err)
}
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-flight port availability check
// for _, p := range []int{7233, 7234, 7235, 7239} {
// 	ln, err := net.Listen("tcp", fmt.Sprintf(":%d", p))
// 	if err != nil { log.Fatalf("port %d in use", p) }
// 	ln.Close()
// }

Try / catch

if err := srv.Start(ctx); err != nil {
	log.Fatalf("one or more services failed to start: %v", err)
	// message lists every failing service via multierr —
	// address the shared root cause if multiple services failed
}

Prevention

When it happens

Trigger: temporal.Server.Start(ctx) -> startServices, when any service's app.Start(ctx) returns an error: failing to bind its gRPC port, failing to join the membership (ringpop/membership host:port unreachable), failing to initialize the service's own persistence/visibility clients, or exceeding the start timeout (max(serviceStartTimeout, 2*MaxJoinDuration)).

Common situations: Port already in use (another Temporal process or stale container holds the frontend/history/matching port); membership advertised IP unreachable in containerized/K8s setups (wrong broadcast host); multiple services sharing a config that points at an unavailable dependency; partial cluster outage where one service can't reach the DB or Elasticsearch.

Related errors


AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/ad026f348ff44e5e. Report an issue: GitHub.

Appendix: source

Thrown at temporal/server_impl.go:142

}

func (s *ServerImpl) startServices() error {
	// The membership join time may exceed the configured max join duration.
	// Double the service start timeout to make sure there is enough time for start logic.
	timeout := max(serviceStartTimeout, 2*s.so.config.Global.Membership.MaxJoinDuration)
	ctx, cancel := context.WithTimeout(context.Background(), timeout)
	defer cancel()

	svcs := slices.Clone(s.servicesMetadata)
	slices.SortFunc(svcs, func(a, b *ServicesMetadata) int {
		return cmp.Compare(initOrder[a.serviceName], initOrder[b.serviceName])
	})

	var allErrs error
	for _, svc := range svcs {
		err := svc.app.Start(ctx)
		if err != nil {
			allErrs = multierr.Append(allErrs, fmt.Errorf("failed to start service %v: %w", svc.serviceName, err))
		}
	}
	return allErrs
}

func initSystemNamespaces(
	ctx context.Context,
	cfg *config.Persistence,
	currentClusterName string,
	persistenceServiceResolver resolver.ServiceResolver,
	persistenceFactoryProvider persistenceClient.FactoryProviderFn,
	logger log.Logger,
	customDataStoreFactory persistenceClient.AbstractDataStoreFactory,
	metricsHandler metrics.Handler,
	serializer serialization.Serializer,
) error {
	clusterName := persistenceClient.ClusterName(currentClusterName)
	metricsHandler = metricsHandler.WithTags(metrics.ServiceNameTag(primitives.ServerService))

View on GitHub (pinned to bde624efd1)