zhisheng17/flink-learning · error · IOException

Failed to set {} state in ConfigMap {} for job {}

Error message

Failed to set {} state in ConfigMap {} for job {}

What it means

writeJobStatusToConfigMap() wraps any failure from the ConfigMap read-modify-write that sets a job's status (Running or Finished) into this IOException. The key write is rejected or the Kubernetes update fails, so the HA registry cannot record the new scheduling state.

Source

Thrown at flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesRunningJobsRegistry.java:120

	private void writeJobStatusToConfigMap(JobID jobID, JobSchedulingStatus status) throws IOException {
		LOG.debug("Setting scheduling state for job {} to {}.", jobID, status);
		final String key = getKeyForJobId(jobID);
		try {
			kubeClient.checkAndUpdateConfigMap(
				configMapName,
				configMap -> {
					if (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {
						final Optional<JobSchedulingStatus> optional = getJobStatus(configMap, jobID);
						if (!optional.isPresent() || optional.get() != status) {
							configMap.getData().put(key, status.name());
							return Optional.of(configMap);
						}
					}
					return Optional.empty();
				}
			).get();
		} catch (Exception e) {
			throw new IOException("Failed to set " + status.name() + " state in ConfigMap "
				+ configMapName + " for job " + jobID, e);
		}
	}

	private Optional<JobSchedulingStatus> getJobStatus(KubernetesConfigMap configMap, JobID jobId) {
		final String key = getKeyForJobId(jobId);
		final String status = configMap.getData().get(key);
		if (!StringUtils.isNullOrWhitespaceOnly(status)) {
			return Optional.of(JobSchedulingStatus.valueOf(status));
		}
		return Optional.empty();
	}

	private String getKeyForJobId(JobID jobId) {
		return RUNNING_JOBS_REGISTRY_KEY_PREFIX + jobId.toString();
	}
}

View on GitHub (pinned to d731cee761)

Solutions

1. Verify leadership and lockIdentity are current; retry setJobRunning/setJobFinished after re-acquiring leadership.
2. Check RBAC: the Flink service account needs get/update on ConfigMaps in the HA namespace.
3. Check the wrapped cause for 409 conflicts and retry with backoff.
4. Confirm the HA ConfigMap exists and is not immutable (Immutable ConfigMaps reject updates).

Example fix

// before: update fails with immutable configmap
kubectl patch cm flink-cluster-1 -p '{"data":{"job-0001":"RUNNING"}}'
// after: remove immutable flag
kubectl patch cm flink-cluster-1 --type merge -p '{"immutable": false}'

// verify write access before registering job state
kubeClient.getConfigMap(haConfigMapName).orElseThrow(
    () -> new IllegalStateException("HA ConfigMap does not exist"));
// and RBAC: kubectl auth can-i update configmaps -n <ns> --as=system:serviceaccount:<ns>:<flink-sa>

try {
    registry.setJobFinished(jobID);
} catch (IOException e) {
    if (ExceptionUtils.findThrowable(e, KubernetesException.class).isPresent()) {
        // re-acquire leadership then retry
    }
}

When it happens

Trigger: setJobRunning(jobID) or setJobFinished(jobID) invoked when the ConfigMap is missing, the caller lost leadership (lock identity mismatch), the API server returns an error, or a resourceVersion conflict occurs because another leader wrote concurrently.

Common situations: Split-brain / stale leader trying to mark a job finished; Kubernetes API throttling during cluster-wide job restarts; ConfigMap made immutable; namespace RBAC forbidding ConfigMap updates.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Failed to clear job state in ConfigMap {} for job {}
• {} already exists in ConfigMap {}
• Could not find {} in ConfigMap {}
• ConfigMap {configMapName} does not exists. It may be deleted
• Failed to update ConfigMap {} since current KubernetesCheckp

AI-assisted analysis of zhisheng17/flink-learning@d731cee761 (2026-09-06). Data as JSON: /api/errors/ff7fa2b56a62f631. Report an issue: GitHub.