What causes "addon %q manifest hash was not populated" in kubernetes/kops?

Ensure AddonManifest.Normalize runs (and reaches the end) for every addon before BootstrapChannelBuilder.Normalize; the hash is set there

kubernetes/kops · error

addon %q manifest hash was not populated

Error message

addon %q manifest hash was not populated

What it means

In the same loop, after confirming addonSpec exists, BootstrapChannelBuilder.Normalize requires addonSpec.ManifestHash to be non-empty. The hash is what lets the channels controller detect manifest changes; an empty hash means AddonManifest.Normalize never completed (hashing happens at its end) or the hash step was skipped, so the channel would be built with an unverifiable addon.

Source

Thrown at upup/pkg/fi/cloudup/bootstrapchannelbuilder/bootstrapchannel.go:63

func (a *BootstrapChannel) GetDependencies(tasks map[string]fi.CloudupTask) []fi.CloudupTask {
	dependencies := make([]fi.CloudupTask, 0, len(a.addonManifests))
	for _, manifest := range a.addonManifests {
		dependencies = append(dependencies, manifest)
	}
	return dependencies
}

func (a *BootstrapChannel) Normalize(c *fi.CloudupContext) error {
	addonsObject := &channelsapi.Addons{}
	addonsObject.Kind = "Addons"
	addonsObject.ObjectMeta.Name = "bootstrap"

	for _, manifest := range a.addonManifests {
		if manifest.addonSpec == nil {
			return fmt.Errorf("addon manifest %q did not have a spec", fi.ValueOf(manifest.Name))
		}
		if manifest.addonSpec.ManifestHash == "" {
			return fmt.Errorf("addon %q manifest hash was not populated", fi.ValueOf(manifest.addonSpec.Name))
		}
		addonsObject.Spec.Addons = append(addonsObject.Spec.Addons, manifest.addonSpec)
	}

	if err := addonsObject.Verify(); err != nil {
		return err
	}

	addonsYAML, err := utils.YamlMarshal(addonsObject)
	if err != nil {
		return fmt.Errorf("error serializing addons yaml: %v", err)
	}

	a.Contents = fi.NewBytesResource(addonsYAML)
	return nil
}

func (a *BootstrapChannel) Find(c *fi.CloudupContext) (*BootstrapChannel, error) {

View on GitHub (pinned to 4c8573c808)

Solutions

Ensure AddonManifest.Normalize runs (and reaches the end) for every addon before BootstrapChannelBuilder.Normalize; the hash is set there
Check for any early return/error swallowed upstream — the addon's own Normalize may have failed silently and left the hash empty
In custom code, never hand-set addonSpec into the channel; always route through the AddonManifest pipeline
Re-run with kops update --dry-run -v to trace whether the per-manifest Normalize executes for the failing addon

Example fix

// before
addonsObject.Spec.Addons = append(addonsObject.Spec.Addons, spec) // hash never computed
// after
m := buildAddonManifest(b, name, spec, res)
if err := m.Normalize(ctx); err != nil { return err }
addonsObject.Spec.Addons = append(addonsObject.Spec.Addons, m.addonSpec)

Defensive patterns

Strategy: validation

Validate before calling

for _, m := range a.addonManifests {
    if m.addonSpec != nil && m.addonSpec.ManifestHash == "" {
        return fmt.Errorf("addon %s: run AddonManifest.Normalize to compute ManifestHash", fi.ValueOf(m.addonSpec.Name))
    }
}

Type guard

func manifestHashed(m *AddonManifest) bool { return m.addonSpec != nil && m.addonSpec.ManifestHash != "" }

Prevention

Never insert addonSpec objects into the channel directly; always go through AddonManifest.Normalize which sets ManifestHash
Check for swallowed errors from per-manifest Normalize during updates
Trace with kops update -v that each addon's Normalize completes before channel aggregation

When it happens

Trigger: Normalize iterates a.addonManifests and finds an entry with a non-nil addonSpec but empty ManifestHash — i.e. the manifest's own Normalize did not run to completion (aborted earlier, was not invoked, or skip paths bypassed hashing) before the channel builder aggregated the specs.

Common situations: Custom kOps builds that register addon specs directly into the channel without running AddonManifest.Normalize; a skipped/failed hashing step upstream that did not abort the whole run; refactors that decouple spec creation from manifest normalization.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/3a19a305786ba04a. Report an issue: GitHub.