kubernetes/kops · critical

serving readiness probe: %w

Error message

serving readiness probe: %w

What it means

In daemon mode, runApplyChannelLoop first calls serveReadiness to start an HTTP readiness endpoint reporting the last apply outcome. If that server cannot be created (e.g. it cannot bind its listen address), the loop aborts immediately with this wrapped error instead of retrying.

Source

Thrown at channels/pkg/cmd/apply_channel.go:130

		}
	}
	if err := RunApplyChannel(ctx, f, out, options, args); err != nil {
		merr = multierr.Append(merr, err)
	}
	return merr
}

// runApplyChannelLoop reconciles repeatedly until ctx is cancelled. A fresh
// ChannelsFactory per iteration drops cached REST configs and the discovery
// cache, picking up cert rotation and new CRDs without a restart.
func runApplyChannelLoop(ctx context.Context, out io.Writer, options *ApplyChannelOptions, args []string) error {
	// In daemon mode kops-channels runs as a system-node-critical static pod; serve a
	// readiness probe reporting the last apply outcome, so a persistent failure surfaces
	// as NotReady (failing `kops validate cluster`, which gates rolling updates) instead
	// of only being logged. Starts NotReady until the first successful apply.
	readiness, err := serveReadiness(ctx)
	if err != nil {
		return fmt.Errorf("serving readiness probe: %w", err)
	}

	// Retry quickly until the first success: the apiserver is usually
	// unreachable while the control plane is still coming up.
	const startupRetryInterval = 5 * time.Second

	settled := false
	for {
		interval := options.Interval
		err := runApplyChannelIteration(ctx, NewChannelsFactory(), out, options, args)
		readiness.recordApplyResult(err)
		if err != nil {
			if !settled {
				interval = min(startupRetryInterval, options.Interval)
			}
			klog.Warningf("error in apply iteration (will retry in %s): %v", interval, err)
		} else {
			settled = true

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check for and kill any existing kops-channels process holding the readiness port (ss -ltnp / lsof)
2. Fix the static pod spec so the readiness port is unique and not conflicting with hostPort mappings
3. Restart the pod/container so the network namespace is clean
4. If serveReadiness supports an address option, bind to a free port

// check the readiness port is free before starting the daemon
ln, err := net.Listen("tcp", readinessAddr)
if err != nil {
	return fmt.Errorf("readiness port %s unavailable: %w", readinessAddr, err)
}
ln.Close()

When it happens

Trigger: serveReadiness fails to bind its TCP port: the readiness port is already in use by another process, the address is not assignable in the network namespace, or insufficient permissions to listen.

Common situations: Two kops-channels daemon instances running concurrently (e.g. leftover process during pod restart); a hostPort/port collision in the static pod spec; restricted environment blocking listen().

Related errors

• error listening on %q: %v
• unexpected return from ListenAndServe

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/e61a4a1da4cbe91d. Report an issue: GitHub.