k3s-io/k3s · error

default runtime %s was not found

Error message

default runtime %s was not found

What it means

On Linux, the agent temporarily prepends its bundled runtimes directory to PATH, discovers available container runtimes via findContainerRuntimes(), then verifies cfg.DefaultRuntime (from --default-runtime / containerd config) is among them. If the name is not in the discovered set (and non-empty), it fails fast with this error before the containerd config is written.

Source

Thrown at pkg/agent/containerd/config_linux.go:83

	if disableCgroup {
		logrus.Warn("cgroup v2 controllers are not delegated for rootless. Disabling cgroup.")
	} else {
		// note: this mutatation of the passed agent.Config is later used to set the
		// kubelet's cgroup-driver flag. This may merit moving to somewhere else in order
		// to avoid mutating the configuration while setting up containerd.
		cfg.AgentConfig.Systemd = !isRunningInUserNS && controllers["cpuset"] && os.Getenv("INVOCATION_ID") != ""
	}

	// set the path to include the default runtimes and remove the aditional path entries
	// that we added after finding the runtimes
	originalPath := os.Getenv("PATH")
	os.Setenv("PATH", runtimesPath+string(os.PathListSeparator)+originalPath)
	extraRuntimes := findContainerRuntimes()
	os.Setenv("PATH", originalPath)

	// Verifies if the DefaultRuntime can be found
	if _, ok := extraRuntimes[cfg.DefaultRuntime]; !ok && cfg.DefaultRuntime != "" {
		return fmt.Errorf("default runtime %s was not found", cfg.DefaultRuntime)
	}

	containerdConfig := templates.ContainerdConfig{
		NodeConfig:            cfg,
		DisableCgroup:         disableCgroup,
		SystemdCgroup:         cfg.AgentConfig.Systemd,
		IsRunningInUserNS:     isRunningInUserNS,
		EnableUnprivileged:    kernel.CheckKernelVersion(4, 11, 0),
		NonrootDevices:        cfg.Containerd.NonrootDevices,
		PrivateRegistryConfig: cfg.AgentConfig.Registry,
		ExtraRuntimes:         extraRuntimes,
		Program:               version.Program,
		NoDefaultEndpoint:     cfg.Containerd.NoDefault,
	}

	selEnabled, selConfigured, err := selinuxStatus()
	if err != nil {
		return fmt.Errorf("failed to detect selinux: %w", err)

View on GitHub (pinned to 6ba341e396)

Solutions

1. Install the runtime on the node (e.g. nvidia-container-toolkit) and confirm with 'which nvidia-container-runtime'
2. Match the exact runtime name that discovery expects (see the error text)
3. Drop --default-runtime to fall back to containerd's default (runc)

Example fix

# before
--default-runtime nvidia   # toolkit not installed

# after
apt-get install -y nvidia-container-toolkit
--default-runtime nvidia   # now discoverable on PATH

if cfg.DefaultRuntime != "" {
    if _, err := exec.LookPath(cfg.DefaultRuntime); err != nil {
        return fmt.Errorf("default runtime %s not installed on this node", cfg.DefaultRuntime)
    }
}

When it happens

Trigger: --default-runtime=nvidia (or crun, kata, ...) on a node where that runtime binary is not installed or not on the effective PATH; or a name mismatch with the discovered key (e.g. 'nvidia' vs 'nvidia-container-runtime').

Common situations: GPU nodes without nvidia-container-toolkit installed; runtimes installed in a nonstandard directory not on PATH; config copied from a GPU node onto a CPU node.

Related errors

• embedded registry mirror requires embedded containerd
• nix-store not found in PATH: install nix (https://nixos.org/
• invalid endpoint URL %s for %s: %v
• failed to detect selinux: %w
• insufficient PSK bytes

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/a7e73173a9dba4a3. Report an issue: GitHub.