SigNoz/signoz · critical

ErrCodeServiceFailed

ErrCodeServiceFailed

Error message

service %q failed before becoming healthy

What it means

Thrown by the service registry's AwaitHealthy when a service's Start() returned an error (startErr) before the healthy signal fired. It distinguishes a startup failure from a silent termination (which gets the 'terminated' variant), wrapping the original start error with ErrCodeServiceFailed.

Source

Thrown at pkg/factory/registry.go:184

		}
	}

	return errors.Join(errs...)
}

// AwaitHealthy blocks until all services reach the RUNNING state or any service fails.
func (registry *Registry) AwaitHealthy(ctx context.Context) error {
	for _, ss := range registry.services {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-ss.healthyC:
		case <-ss.startReturnedC:
			ss.mu.RLock()
			err := ss.startErr
			ss.mu.RUnlock()
			if err != nil {
				return errors.Wrapf(err, errors.TypeInternal, ErrCodeServiceFailed, "service %q failed before becoming healthy", ss.service.Name())
			}
			return errors.Newf(errors.TypeInternal, ErrCodeServiceFailed, "service %q terminated before becoming healthy", ss.service.Name())
		}
	}
	return nil
}

// ServicesByState returns a snapshot of the current state of all services.
func (registry *Registry) ServicesByState() map[State][]Name {
	result := make(map[State][]Name)
	for _, ss := range registry.services {
		state := ss.getState()
		result[state] = append(result[state], ss.service.Name())
	}
	return result
}

// IsHealthy returns true if all services are in the RUNNING state.

View on GitHub (pinned to 5069bf80b0)

Solutions

1. Read the wrapped startErr — it contains the actual startup failure; fix that first
2. Check for port conflicts and required dependency availability before starting the service
3. Use DependsOn/dependency ordering so prerequisites become healthy first
4. Add startup retries/backoff for dependency connection failures at boot

Example fix

// before
if err := ss.AwaitHealthy(ctx); err != nil { panic(err) }

// after
if err := ss.AwaitHealthy(ctx); err != nil {
	if errors.Is(err, registry.ErrCodeServiceFailed) {
		log.Errorw("service failed to start", "service", name, "startErr", errors.Unwrap(err))
	}
	return err
}

// pre-flight: ensure dependencies are reachable before Start()
if err := db.Ping(); err != nil { return err }

if err := ss.AwaitHealthy(ctx); err != nil {
	return fmt.Errorf("bootstrap failed: %w", err) // wrapped startErr has the cause
}

When it happens

Trigger: Calling AwaitHealthy(ctx) on a registered service whose Start() blocked, then returned an error — e.g. a HTTP server failing to bind its port, a worker failing to connect to its broker, or a migrator failing before signaling healthy.

Common situations: Port already in use (EADDRINUSE) for the service's listener; required dependency (DB, Kafka) unavailable at boot; invalid service configuration causing immediate start failure; startup ordering issues where a dependent service starts before its dependency is healthy.

Related errors

• invalid_input
• invalid_input

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/afcad7b19593209c. Report an issue: GitHub.