zhisheng17/flink-learning · error · KubernetesException

Failed to update ConfigMap {} since current KubernetesCheckp

Error message

Failed to update ConfigMap {} since current KubernetesCheckpointIDCounter does not have the leadership.

What it means

KubernetesCheckpointIDCounter.getAndIncrement performs a compare-and-swap update of the checkpoint counter ConfigMap. If the update reports 'not updated', the current instance has lost leadership, so the counter refuses to hand out a checkpoint ID and throws KubernetesException.

Source

Thrown at flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesCheckpointIDCounter.java:111

	public long getAndIncrement() throws Exception {
		final AtomicLong current = new AtomicLong();
		final boolean updated = kubeClient.checkAndUpdateConfigMap(
			configMapName,
			configMap -> {
				if (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {
					final long currentValue = getCurrentCounter(configMap);
					current.set(currentValue);
					configMap.getData().put(CHECKPOINT_COUNTER_KEY, String.valueOf(currentValue + 1));
					return Optional.of(configMap);
				}
				return Optional.empty();
			}
		).get();

		if (updated) {
			return current.get();
		} else {
			throw new KubernetesException("Failed to update ConfigMap " + configMapName +
				" since current KubernetesCheckpointIDCounter does not have the leadership.");
		}
	}

	@Override
	public long get() {
		return kubeClient.getConfigMap(configMapName)
			.map(this::getCurrentCounter)
			.orElseThrow(() -> new FlinkRuntimeException(
				new KubernetesException("ConfigMap " + configMapName + " does not exist.")));
	}

	@Override
	public void setCount(long newCount) throws Exception {
		kubeClient.checkAndUpdateConfigMap(
			configMapName,
			configMap -> {
				if (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {

View on GitHub (pinned to d731cee761)

Solutions

1. Verify current leadership before triggering checkpoints; after losing it, stop checkpoint initiation and let the new leader take over.
2. Check HA ConfigMap/lease state: kubectl get configmap,leases in the namespace; confirm only one JobManager holds the lease.
3. Reduce checkpoint trigger overlap with failovers (adjust checkpoint interval / postpone triggering during leader transition).
4. Investigate Kubernetes API latency or watch disconnections that cause lease renewal failures.
5. Restart the deposed JobManager so it re-acquires or steps down cleanly instead of operating with stale leadership.

Example fix

// before: trigger checkpoints regardless of leadership
CompletableFuture<CompletedCheckpoint> cc = checkpointCoordinator.triggerCheckpoint(false);
// after: guard on leadership
if (hasLeadership.get()) {
    checkpointCoordinator.triggerCheckpoint(false);
} // else skip; new leader will checkpoint

// only trigger checkpoints while leadership is held
if (!leaderElector.hasLeadership()) return; // skip triggering

try {
    long id = checkpointIDCounter.getAndIncrement();
} catch (KubernetesException e) {
    if (e.getMessage().contains("does not have the leadership")) {
        // stop checkpointing; wait for/fetch new leader state
    }
}

When it happens

Trigger: Calling getAndIncrement (during checkpoint triggering) when this JobManager no longer holds the leader lease — its ConfigMap CAS update fails because another (new leader) instance modified the counter first.

Common situations: Leader change/JobManager failover racing with an in-flight checkpoint trigger; HA lease flapping due to Kubernetes API latency; two JobManagers briefly believing they are leader; ConfigMap update conflicts from concurrent resource-version changes.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Could not find {key} in ConfigMap {configMapName}
• ConfigMap {configMapName} does not exists. It may be deleted
• Could not get the rest endpoint of ${clusterId}
• Could not create the RestClusterClient.
• The Flink cluster ${clusterId} already exists.

AI-assisted analysis of zhisheng17/flink-learning@d731cee761 (2026-09-06). Data as JSON: /api/errors/54065a7d309296e1. Report an issue: GitHub.