{"record":{"id":"22b7d7c1383f250d","repo":"kubernetes/kops","slug":"unable-to-find-any-containerd-binaries-in-assets","errorCode":null,"errorMessage":"unable to find any containerd binaries in assets","messagePattern":"unable to find any containerd binaries in assets","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nodeup/pkg/model/containerd.go","lineNumber":127,"sourceCode":"var ContainerdLicense string\n\n// installContainerd installs the binaries and services to run containerd.\n// We break it out because on immutable OSes we only configure containerd, we don't install it.\nfunc (b *ContainerdBuilder) installContainerd(c *fi.NodeupModelBuilderContext) error {\n\t// Add Apache2 license\n\t{\n\t\tt := &nodetasks.File{\n\t\t\tPath:     \"/usr/share/doc/containerd/apache.txt\",\n\t\t\tContents: fi.NewStringResource(ContainerdLicense),\n\t\t\tType:     nodetasks.FileType_File,\n\t\t}\n\t\tc.AddTask(t)\n\t}\n\n\t// Add containerd binaries from containerd release package\n\tf := b.Assets.FindMatches(regexp.MustCompile(`^bin/(containerd|ctr)`))\n\tif len(f) == 0 {\n\t\treturn fmt.Errorf(\"unable to find any containerd binaries in assets\")\n\t}\n\tfor k, v := range f {\n\t\tfileTask := &nodetasks.File{\n\t\t\tPath:     filepath.Join(\"/usr/bin\", k),\n\t\t\tContents: v,\n\t\t\tType:     nodetasks.FileType_File,\n\t\t\tMode:     new(\"0755\"),\n\t\t}\n\t\tc.AddTask(fileTask)\n\t}\n\n\t// Add runc binary from https://github.com/opencontainers/runc\n\t// https://github.com/containerd/containerd/issues/6541\n\tf = b.Assets.FindMatches(regexp.MustCompile(`/runc\\.(amd64|arm64)$`))\n\tif len(f) != 1 {\n\t\treturn fmt.Errorf(\"error finding runc asset\")\n\t}\n\tfor _, v := range f {","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/nodeup/pkg/model/containerd.go#L109-L145","documentation":"nodeup's ContainerdBuilder.installContainerd looks through the node's bundled asset archive for files matching ^bin/(containerd|ctr) (the containerd release binaries). If the archive contains none of them, nodeup cannot install containerd on the node and fails fast with this error rather than producing a broken node. It is thrown at nodeup/pkg/model/containerd.go:127 during node bootstrap.","triggerScenarios":"Running nodeup with installContainerd enabled while the nodeup asset set contains no files named bin/containerd or bin/ctr — typically because the containerd release package was not attached to the cluster spec for that architecture/OS, or a kOps upgrade changed the expected asset names/paths.","commonSituations":"Custom containerd asset URLs (ContainerdConfig.Version pointing at a version with no matching binary bundled), air-gapped clusters where assets were mirrored incompletely, mixing architectures (e.g. arm64 node with amd64-only assets), or a broken/partial kOps upgrade where newer nodeup runs against older cluster configuration.","solutions":["Check the nodeup config/asset archive for bin/containerd and bin/ctr; ensure the containerd release package for the node's architecture is included","Set/verify spec.containerd.version in the cluster spec to a kOps-supported containerd version so kOps bundles the correct assets","Run `kops update cluster` (and `kops rolling-update cluster`) so nodes get a nodeup config matching the current kOps version","For custom/air-gapped installs, verify the mirrored asset store includes the full containerd release archive"],"exampleFix":"// before (cluster.yaml)\ncontainerd:\n  version: 1.7.99   # unsupported, kOps has no bundled binary\n// after\ncontainerd:\n  version: 1.7.18   # kOps-supported version with bundled bin/containerd + bin/ctr assets","handlingStrategy":"validation","validationCode":"# Before running nodeup, verify the asset archive contains the containerd binaries:\n# e.g. if assets are in a tarball:\ntar -tf nodeup-assets.tar.gz | grep -E '^bin/(containerd|ctr)'\n# exit code 0 and at least one match means nodeup will proceed","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin spec.containerd.version to a kOps-supported release so assets are bundled automatically","Re-run `kops update cluster` after every kOps upgrade before rolling nodes","Verify mirrored asset stores contain bin/containerd and bin/ctr for every node architecture","Match node OS/arch to the asset set you provide"],"tags":["nodeup","containerd","assets"],"backgroundTag":"missing-asset-in-bundle","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}