kubernetes/kops · critical

unable to find any containerd binaries in assets

Error message

unable to find any containerd binaries in assets

What it means

nodeup's ContainerdBuilder.installContainerd looks through the node's bundled asset archive for files matching ^bin/(containerd|ctr) (the containerd release binaries). If the archive contains none of them, nodeup cannot install containerd on the node and fails fast with this error rather than producing a broken node. It is thrown at nodeup/pkg/model/containerd.go:127 during node bootstrap.

Source

Thrown at nodeup/pkg/model/containerd.go:127

var ContainerdLicense string

// installContainerd installs the binaries and services to run containerd.
// We break it out because on immutable OSes we only configure containerd, we don't install it.
func (b *ContainerdBuilder) installContainerd(c *fi.NodeupModelBuilderContext) error {
	// Add Apache2 license
	{
		t := &nodetasks.File{
			Path:     "/usr/share/doc/containerd/apache.txt",
			Contents: fi.NewStringResource(ContainerdLicense),
			Type:     nodetasks.FileType_File,
		}
		c.AddTask(t)
	}

	// Add containerd binaries from containerd release package
	f := b.Assets.FindMatches(regexp.MustCompile(`^bin/(containerd|ctr)`))
	if len(f) == 0 {
		return fmt.Errorf("unable to find any containerd binaries in assets")
	}
	for k, v := range f {
		fileTask := &nodetasks.File{
			Path:     filepath.Join("/usr/bin", k),
			Contents: v,
			Type:     nodetasks.FileType_File,
			Mode:     new("0755"),
		}
		c.AddTask(fileTask)
	}

	// Add runc binary from https://github.com/opencontainers/runc
	// https://github.com/containerd/containerd/issues/6541
	f = b.Assets.FindMatches(regexp.MustCompile(`/runc\.(amd64|arm64)$`))
	if len(f) != 1 {
		return fmt.Errorf("error finding runc asset")
	}
	for _, v := range f {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check the nodeup config/asset archive for bin/containerd and bin/ctr; ensure the containerd release package for the node's architecture is included
2. Set/verify spec.containerd.version in the cluster spec to a kOps-supported containerd version so kOps bundles the correct assets
3. Run `kops update cluster` (and `kops rolling-update cluster`) so nodes get a nodeup config matching the current kOps version
4. For custom/air-gapped installs, verify the mirrored asset store includes the full containerd release archive

Example fix

// before (cluster.yaml)
containerd:
  version: 1.7.99   # unsupported, kOps has no bundled binary
// after
containerd:
  version: 1.7.18   # kOps-supported version with bundled bin/containerd + bin/ctr assets

# Before running nodeup, verify the asset archive contains the containerd binaries:
# e.g. if assets are in a tarball:
tar -tf nodeup-assets.tar.gz | grep -E '^bin/(containerd|ctr)'
# exit code 0 and at least one match means nodeup will proceed

When it happens

Trigger: Running nodeup with installContainerd enabled while the nodeup asset set contains no files named bin/containerd or bin/ctr — typically because the containerd release package was not attached to the cluster spec for that architecture/OS, or a kOps upgrade changed the expected asset names/paths.

Common situations: Custom containerd asset URLs (ContainerdConfig.Version pointing at a version with no matching binary bundled), air-gapped clusters where assets were mirrored incompletely, mixing architectures (e.g. arm64 node with amd64-only assets), or a broken/partial kOps upgrade where newer nodeup runs against older cluster configuration.

Related errors

• error finding runc asset
• error finding contained version
• error parsing container runtime version %q: %w
• error building containerd flags: %v
• building json: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/22b7d7c1383f250d. Report an issue: GitHub.