What causes "error running kubectl: %v" in kubernetes/kops?

Check the klog output immediately above the error — kops logs the exact command and kubectl's combined output

error running kubectl: %v

Error message

error running kubectl: %v

What it means

execKubectl runs a kubectl subprocess with CombinedOutput and wraps any non-zero exit into `error running kubectl: %v`, returning the combined stdout/stderr as the string result and logging both the command and output. It is the low-level executor behind all kubectl invocations in KubectlApplier (apply/replace).

Source

Thrown at channels/pkg/channels/kubectlapplier.go:91

		}
	}

	return nil
}

func execKubectl(ctx context.Context, args ...string) (string, error) {
	kubectlPath := "kubectl" // Assume in PATH
	cmd := exec.CommandContext(ctx, kubectlPath, args...)
	env := os.Environ()
	cmd.Env = env

	human := strings.Join(cmd.Args, " ")
	klog.V(2).Infof("Running command: %s", human)
	output, err := cmd.CombinedOutput()
	if err != nil {
		klog.Infof("error running %s", human)
		klog.Info(string(output))
		return string(output), fmt.Errorf("error running kubectl: %v", err)
	}

	return string(output), err
}

View on GitHub (pinned to 4c8573c808)

Solutions

Check the klog output immediately above the error — kops logs the exact command and kubectl's combined output
Run the logged kubectl command manually to reproduce and see the raw message
Install kubectl or fix PATH so `kubectl` resolves
Fix kubeconfig/context/network so kubectl can reach the API server
Resolve the underlying API error (RBAC role bindings, missing resources, manifest validation)

Example fix

// before (CI image without kubectl)
FROM alpine
RUN kops apply ...
// after
FROM alpine
RUN apk add --no-cache curl && curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && install kubectl /usr/local/bin/

Defensive patterns

Strategy: try-catch

Validate before calling

// before invoking apply: check kubectl and connectivity
if _, err := exec.LookPath("kubectl"); err != nil {
    return fmt.Errorf("kubectl missing from PATH: %w", err)
}
out, err := exec.Command("kubectl", "version").CombinedOutput()
if err != nil {
    return fmt.Errorf("kubectl cannot reach cluster: %s", out)
}

Try / catch

out, err := execKubectl(ctx, args...)
if err != nil {
    // execKubectl already logged command + combined output
    return fmt.Errorf("kubectl %v: %s: %w", args, out, err)
}

Prevention

Install kubectl in every environment that runs kops update
Set KUBECONFIG explicitly rather than relying on default paths
Run `kubectl auth can-i --list` to verify RBAC before updates
Capture klog output (the command and combined output are logged on failure)

When it happens

Trigger: Any `kubectl apply`/`kubectl replace` invocation exits non-zero: kubectl binary not found in PATH, exec failure, or kubectl reporting an API error (403, 404, validation failure, connection refused).

Common situations: kubectl not installed in the container/host running kops; KUBECONFIG pointing at the wrong cluster; API server unreachable; RBAC forbidden; `replace` failing on clusterIP: "" on some k8s versions (logged but not returned, unlike the final apply).

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/4462ee2249c1dc60. Report an issue: GitHub.