kubernetes/kops · warning

error executing command %q: %v

Error message

error executing command %q: %v

What it means

Returned by logDumperNode.shellToFile (pkg/dump/dumper.go:577) when the remote command executed via n.client.ExecPiped fails after the local destination file was created successfully. ExecPiped returns ctx.Err() if the dump context already expired or is cancelled, the SSH session creation error from NewSession, or session.Run's non-zero-exit error. The local file is left behind (possibly empty/partial), and dump only logs this as a warning.

Source

Thrown at pkg/dump/dumper.go:577

		services = append(services, tokens[0])
	}
	return services, nil
}

// shellToFile executes a command and copies the output to a file
func (n *logDumperNode) shellToFile(ctx context.Context, command string, destPath string) error {
	if err := os.MkdirAll(filepath.Dir(destPath), 0o755); err != nil {
		klog.Warningf("unable to mkdir on %q: %v", filepath.Dir(destPath), err)
	}

	f, err := os.Create(destPath)
	if err != nil {
		return fmt.Errorf("error creating file %q: %v", destPath, err)
	}
	defer f.Close()

	if err := n.client.ExecPiped(ctx, command, f, f); err != nil {
		return fmt.Errorf("error executing command %q: %v", command, err)
	}

	return nil
}

// sshClientImplementation is the default implementation of sshClient, binding to a *ssh.Client
type sshClientImplementation struct {
	client    *ssh.Client
	forwardTo string
}

var _ sshClient = &sshClientImplementation{}

// ExecPiped implements sshClientImplementation::ExecPiped
func (s *sshClientImplementation) ExecPiped(ctx context.Context, cmd string, stdout io.Writer, stderr io.Writer) error {
	if ctx.Err() != nil {
		return ctx.Err()
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Run the failing command (named in the error) manually over SSH to see the remote stderr and exit code.
2. If the command binary is missing on the image (nft, conntrack, journalctl, kubectl), treat as expected for that image or install/probe for it.
3. Raise --node-dump-timeout for large logs or slow nodes; dumps are sequential so budget is shared.
4. If wrapped error is 'error creating ssh session', check sshd MaxSessions and reduce concurrent SSH use; retry after transient network resets.
5. Ignore known-benign failures: dumpNode treats these as warnings; only act if the specific artifact you need is empty in the artifacts dir.

Example fix

// before
if err := n.client.ExecPiped(ctx, command, f, f); err != nil {
    return fmt.Errorf("error executing command %q: %v", command, err)
}
// after (guard for absent binaries)
guarded := "command -v " + firstWord(command) + " &>/dev/null && " + command + " || true"
if err := n.client.ExecPiped(ctx, guarded, f, f); err != nil {
    return fmt.Errorf("error executing command %q: %w", command, err)
}

// check command availability remotely before capturing
probe := "command -v " + shellQuote(binary) + " >/dev/null"
if err := n.client.ExecPiped(ctx, probe, io.Discard, io.Discard); err != nil {
    klog.V(2).Infof("skipping %s: binary missing on node", binary)
}

if err := n.client.ExecPiped(ctx, command, f, f); err != nil {
    if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
        klog.Warningf("capture of %q cancelled: %v", command, err)
    } else {
        klog.Warningf("command %q failed (exit!=0 or session error): %v", command, err)
    }
    return fmt.Errorf("error executing command %q: %w", command, err)
}

When it happens

Trigger: ExecPiped(ctx, command, f, f) fails for any captured command (journalctl, iptables, kubectl logs, sudo cat ...): the 1-minute nodeDumpTimeout expires mid-transfer, the SSH session can't be created, the remote command exits non-zero (missing binary like iptables/nft/conntrack, journalctl absent, kubectl not on node), or connection reset.

Common situations: Node images lacking one of the probed commands (nft, conntrack, journalctl); /var/log files too large to transfer within the timeout; kubectl missing on the node for podSelectors; node rebooting or network flapping mid-dump; too many parallel SSH sessions exhausting MaxSessions on sshd.

Related errors

• connecting: %w
• unable to SSH to %q: %v
• error listing systemd services: %v
• error reading /var/log: %v
• error listing /etc/containerd: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/5958b14e5000014f. Report an issue: GitHub.