kubernetes/kops · warning

error creating ssh session: %v

Error message

error creating ssh session: %v

What it means

Produced inside sshClientImplementation.ExecPiped (pkg/dump/dumper.go:601) when s.client.NewSession() fails on the established *ssh.Client connection. The goroutine sends this error on the finished channel; ExecPiped returns it unless the context expires first. It means the SSH transport is still up (or believed to be) but no new channel could be opened — typically a client/server session or resource limit.

Source

Thrown at pkg/dump/dumper.go:601

// sshClientImplementation is the default implementation of sshClient, binding to a *ssh.Client
type sshClientImplementation struct {
	client    *ssh.Client
	forwardTo string
}

var _ sshClient = &sshClientImplementation{}

// ExecPiped implements sshClientImplementation::ExecPiped
func (s *sshClientImplementation) ExecPiped(ctx context.Context, cmd string, stdout io.Writer, stderr io.Writer) error {
	if ctx.Err() != nil {
		return ctx.Err()
	}

	finished := make(chan error)
	go func() {
		session, err := s.client.NewSession()
		if err != nil {
			finished <- fmt.Errorf("error creating ssh session: %v", err)
			return
		}
		defer session.Close()

		session.Stdout = stdout
		session.Stderr = stderr

		if s.forwardTo != "" {
			cmd = fmt.Sprintf("ssh -o 'StrictHostKeyChecking no' %s %s", quoteShell(s.forwardTo), quoteShell(cmd))
		}

		klog.V(2).Infof("running SSH command: %v", cmd)

		finished <- session.Run(cmd)
	}()

	select {
	case <-ctx.Done():

View on GitHub (pinned to 4c8573c808)

Solutions

1. Raise sshd MaxSessions on the node (drop-in in /etc/ssh/sshd_config.d) if many channels are opened concurrently, then restart sshd.
2. Retry the command: reconnect via Dial instead of reusing a possibly-stale client, since NewSession failure often means the transport is dead.
3. Check for idle-connection timeouts (NAT, security appliance, sshd ClientAliveInterval) and enable keepalives on the SSH dialer.
4. Verify the connection wasn't closed early by the caller (n.Close() racing with in-flight dumps).
5. Inspect the wrapped error text — 'connection reset by peer' vs 'session open failed' distinguishes network drops from server-side channel limits.

Example fix

// before
session, err := s.client.NewSession()
if err != nil {
    finished <- fmt.Errorf("error creating ssh session: %v", err)
    return
}
// after (caller-side: re-dial on session failure)
session, err := s.client.NewSession()
if err != nil {
    finished <- fmt.Errorf("error creating ssh session: %w", err)
    return
}

// verify the transport is alive before opening a session
if s.client.Conn() == nil || s.client.Conn().Closed() {
    return fmt.Errorf("ssh transport already closed; redial required")
}

session, err := s.client.NewSession()
if err != nil {
    // retry once after redial
    if client2, dialErr := factory.Dial(ctx, host, useBastion); dialErr == nil {
        if s2, e2 := client2.NewSession(); e2 == nil {
            session = s2
            err = nil
        }
    }
}
if err != nil {
    finished <- fmt.Errorf("error creating ssh session: %w", err)
    return
}

When it happens

Trigger: s.client.NewSession() returns an error: sshd MaxSessions (default 10) exhausted on the target node, the underlying TCP connection was reset/closed by the server or a middlebox, the server refused the channel ('session open failed'), or the client connection was already closed.

Common situations: Dumping many nodes/services in parallel while other tooling holds SSH channels to the same node (MaxSessions hit); NAT/firewall idle timeouts killing the connection between Dial and the command run; bastion-forwarded connections dropping; long dumps where the connection goes stale mid-run.

Related errors

• connecting: %w
• unable to SSH to %q: %v
• error listing systemd services: %v
• error reading /var/log: %v
• error listing /etc/containerd: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/a32f8c305d0ae32f. Report an issue: GitHub.