k3s-io/k3s · error

critical configuration value mismatch between servers

Error message

critical configuration value mismatch between servers

What it means

During join/reconciliation the server deep-compares its config.CriticalControlArgs against CriticalControlArgs stored in the cluster bootstrap. Any diff logs per-field warnings ('critical configuration mismatched: <cli tag>') and returns this error, blocking a server whose fundamental networking/runtime flags disagree with the cluster.

Source

Thrown at pkg/cluster/bootstrap.go:558

		clusterControl.CriticalControlArgs.EgressSelectorMode = c.config.CriticalControlArgs.EgressSelectorMode
	}
	// If the remote server is down-level, for secrets-encryption-key-type
	if clusterControl.CriticalControlArgs.EncryptProvider == "" {
		clusterControl.CriticalControlArgs.EncryptProvider = c.config.CriticalControlArgs.EncryptProvider
	}

	if diff := deep.Equal(c.config.CriticalControlArgs, clusterControl.CriticalControlArgs); diff != nil {
		rc := reflect.ValueOf(clusterControl.CriticalControlArgs).Type()
		for _, d := range diff {
			field := strings.Split(d, ":")[0]
			v, _ := rc.FieldByName(field)
			if cliTag, found := v.Tag.Lookup("cli"); found {
				logrus.Warnf("critical configuration mismatched: %s", cliTag)
			} else {
				logrus.Warnf("critical configuration mismatched: %s", field)
			}
		}
		return errors.New("critical configuration value mismatch between servers")
	}
	return nil
}

// ipsTo16Bytes makes sure the IPs in the []*net.IPNet slice are represented in 16-byte format
func ipsTo16Bytes(mySlice []*net.IPNet) {
	for _, ipNet := range mySlice {
		ipNet.IP = ipNet.IP.To16()
	}
}

View on GitHub (pinned to 6ba341e396)

Solutions

1. Read the 'critical configuration mismatched: <flag>' warning lines immediately above the error; they name the exact disagreeing flags.
2. Edit this node's /etc/rancher/k3s/config.yaml or CLI flags so the listed values match the first server exactly.
3. Restart k3s on this node; if the first server's values were themselves wrong, plan a coordinated cluster reinit instead of divergent flags.

Example fix

# before (joining node)
cluster-cidr: 10.42.0.0/16
service-cidr: 10.43.0.0/16
# after (match the initiating server)
cluster-cidr: 10.52.0.0/16
service-cidr: 10.53.0.0/16

// Before joining, fetch the cluster's critical args (e.g., from /v1-k3s/config or a config map you publish) and diff:
func criticalArgsCompatible(mine, cluster v1critical.Config) []string {
	var diffs []string
	if mine.ClusterCIDR != cluster.ClusterCIDR { diffs = append(diffs, "cluster-cidr") }
	if mine.ServiceCIDR != cluster.ServiceCIDR { diffs = append(diffs, "service-cidr") }
	if mine.ClusterDNS != cluster.ClusterDNS { diffs = append(diffs, "cluster-dns") }
	return diffs // empty = safe to join
}

if err := joinCluster(); err != nil {
	if strings.Contains(err.Error(), "critical configuration value mismatch between servers") {
		// read the preceding 'critical configuration mismatched: <flag>' warnings and align config
	}
	return err
}

When it happens

Trigger: Joining server passes different values for any CriticalControlArgs field (cluster-cidr, service-cidr, cluster-dns, disable-network-policy, flannel backend, etc.) via CLI flags or /etc/rancher/k3s/config.yaml than the values published by the first server.

Common situations: Config file copied from a template with a different cluster-cidr; one node got --cluster-dns manually; nodes in the same cluster built from different install scripts or versions where defaults changed.

Related errors

• etcd-s3-timeout must be greater than 0s
• all cloud-provider functionality disabled by config
• no bootstrap data is available to reconcile against
• bootstrap data already found and encrypted with different to
• this server has not yet been promoted from learner to voting

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/79091121099a41e6. Report an issue: GitHub.