cilium/cilium · error

failed to start kubeproxy healthz server: %w

Error message

failed to start kubeproxy healthz server: %w

What it means

The kube-proxy healthz compatibility server (which mimics kube-proxy's /healthz endpoint) fails to bind its TCP listener for reasons other than the expected 'address not available' case. The hint logged indicates kube-proxy itself is usually the conflicting listener; this error wraps the raw listen error.

Source

Thrown at daemon/healthz/kube_proxy_healthz.go:95

// status HTTP endpoint exposed on addr.
// This endpoint reports the agent health status with the timestamp.
func registerKubeProxyHealthzHTTPService(params kubeProxyHealthParams) error {
	if params.Config.KubeProxyReplacementHealthzBindAddress == "" || !params.KPRConfig.KubeProxyReplacement {
		return nil
	}

	params.JobGroup.Add(job.OneShot("kube-proxy-healthz-server", func(ctx context.Context, health cell.Health) error {
		addr := params.Config.KubeProxyReplacementHealthzBindAddress
		lc := net.ListenConfig{Control: setsockoptReuseAddrAndPort}
		ln, err := lc.Listen(ctx, "tcp", addr)
		if errors.Is(err, unix.EADDRNOTAVAIL) {
			params.Logger.Info("KubeProxy healthz server not available", logfields.Address, addr)
		} else if err != nil {
			params.Logger.Error("hint: kube-proxy should not be running nor listening on the same healthz-bind-address.",
				logfields.Address, addr,
				logfields.Error, err,
			)
			return fmt.Errorf("failed to start kubeproxy healthz server: %w", err)
		}

		mux := http.NewServeMux()
		mux.Handle("/healthz", kubeproxyHealthzHandler{
			statusCollector: params.StatusCollector,
			lastUpdateAter:  params.BPFOps,
			localNode:       params.NodeLocalStore,
		})

		srv := &http.Server{
			Addr:    addr,
			Handler: mux,
		}

		params.Logger.Info("Starting kube-proxy healthz server", logfields.Address, addr)

		ctx, cancel := context.WithCancel(ctx)
		defer cancel()

View on GitHub (pinned to ac7b90affa)

Solutions

1. Stop any real kube-proxy instance running alongside Cilium (kube-proxy replacement) or align the healthz-bind-address
2. Run the agent pod with hostNetwork: true and the required capabilities (NET_BIND_SERVICE) so it can bind 10256
3. Change healthz-bind-address to a free, unprivileged port if the default is unusable
4. Check LSM/firewall policies that block listening on the address

addr := fmt.Sprintf("%s:%d", bindAddr, 10256)
if ln, err := net.Listen("tcp", addr); err == nil { ln.Close() } else if !errors.Is(err, unix.EADDRINUSE) {
	log.WithError(err).Warnf("cannot bind kube-proxy healthz address %s", addr)
}

if err := startKubeProxyHealthzServer(...); err != nil {
	if errors.Is(err, unix.EADDRINUSE) {
		log.Warn("kube-proxy already owns the healthz port; disable kube-proxy or change bind address")
	}
	return err
}

When it happens

Trigger: Listening on the kube-proxy healthz-bind-address (default 0.0.0.0:10256) fails with an unexpected error such as EACCES, EPERM, or invalid address — distinct from the tolerated case where kube-proxy already owns the port.

Common situations: Running cilium in kube-proxy-replacement mode while a leftover kube-proxy still runs and SO_REUSEPORT handling differs; binding a privileged port without capabilities; SELinux policies blocking the bind; hostNetworking restrictions in the agent pod.

Related errors

• healthz status API server not available: %w
• failed to start healthz status API server: %w
• failed to start kube-proxy healthz server: %w
• no healthz status API server started
• failed to setsockopt(SO_REUSEADDR): %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/772b679d3005ca36. Report an issue: GitHub.