kubernetes/kops · critical

nodeup config hash mismatch (was %q, expected %q)

Error message

nodeup config hash mismatch (was %q, expected %q)

What it means

After loading nodeupconfig.yaml, nodeup computes its SHA-256 hash and compares it to the base64-encoded NodeupConfigHash embedded in the node's BootConfig. This is an integrity/consistency check ensuring the node boots with the exact config that was provisioned for it. A mismatch means the file in the state store changed (or the boot config is stale) since userdata was generated.

Source

Thrown at upup/pkg/fi/nodeup/command.go:163

	case bootConfig.InstanceGroupName != "":
		nodeupConfigLocation := configBase.Join("igconfig", bootConfig.InstanceGroupRole.ToLowerString(), bootConfig.InstanceGroupName, "nodeupconfig.yaml")

		b, err := nodeupConfigLocation.ReadFile(ctx)
		if err != nil {
			return fmt.Errorf("error loading NodeupConfig %q: %v", nodeupConfigLocation, err)
		}

		if err = utils.YamlUnmarshal(b, &nodeupConfig); err != nil {
			return fmt.Errorf("error parsing NodeupConfig %q: %v", nodeupConfigLocation, err)
		}
		nodeupConfigHash = sha256.Sum256(b)
	default:
		return fmt.Errorf("no instance group defined in nodeup config")
	}

	if bootConfig.NodeupConfigHash != "" {
		if want, got := bootConfig.NodeupConfigHash, base64.StdEncoding.EncodeToString(nodeupConfigHash[:]); got != want {
			return fmt.Errorf("nodeup config hash mismatch (was %q, expected %q)", got, want)
		}
	}

	err = evaluateSpec(&nodeupConfig, bootConfig.CloudProvider, region)
	if err != nil {
		return err
	}

	architecture, err := architectures.FindArchitecture()
	if err != nil {
		return fmt.Errorf("error determining OS architecture: %v", err)
	}

	distribution, err := distributions.FindDistribution("/")
	if err != nil {
		return fmt.Errorf("error determining OS distribution: %v", err)
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Re-run 'kops update cluster --yes' (and 'kops rolling-update cluster --yes') so userdata/launch templates are regenerated in sync with the current nodeupconfig.yaml, then replace the instance.
2. If the config should not have changed, restore the original nodeupconfig.yaml that matches the boot config hash (from object-store versioning or backup).
3. Ensure the ASG/instance group launches instances with the latest userdata (refresh instance refresh / launch template version).
4. Verify no manual edits were made to igconfig/<role>/<ig>/nodeupconfig.yaml in the state store; edit the kOps cluster spec and re-apply instead.

Example fix

// before: node booting with stale userdata after config change
// instance launches with old launch template
// after: refresh instance group so userdata hash matches config
kops update cluster --yes && kops rolling-update cluster --yes

// Verify the config hash matches before running nodeup
b, _ := nodeupConfigLocation.ReadFile(ctx)
got := base64.StdEncoding.EncodeToString(sha256Sum(b))
if bootConfig.NodeupConfigHash != "" && got != bootConfig.NodeupConfigHash {
    return fmt.Errorf("stale boot config: re-run kops update cluster / refresh userdata")
}

err := cmd.Run(out)
if err != nil && strings.Contains(err.Error(), "nodeup config hash mismatch") {
    // trigger ASG instance refresh so userdata matches current config
}

When it happens

Trigger: Running NodeUpCommand.Run() when bootConfig.NodeupConfigHash is non-empty and the hash of the freshly-read nodeupconfig.yaml differs from it — e.g. the state store's nodeupconfig.yaml was regenerated by a new 'kops update cluster' while the node still boots with old userdata, or the file was edited/replaced manually.

Common situations: Rolling updates in progress: userdata from a previous kops apply plus new config in S3; ASG launch template not refreshed after 'kops update cluster --yes'; manual edits to nodeupconfig.yaml; object-store versioning serving a different object version.

Related errors

• building nodeConfig for instanceGroup: %w
• unsupported cloud provider for authenticator %q
• no keypairID for %q
• challenge not set
• challenge.id not set

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/c6533946817fd6df. Report an issue: GitHub.