zhisheng17/flink-learning · error · IOException

Failed to clear job state in ConfigMap {} for job {}

Error message

Failed to clear job state in ConfigMap {} for job {}

What it means

KubernetesRunningJobsRegistry.clearJob() wraps any exception from its read-modify-write ConfigMap transaction (removing the job's status key) into this IOException. The Kubernetes API call inside the lambda or the blocking .get() failed, so the job's scheduling state could not be cleared from the HA ConfigMap.

Source

Thrown at flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesRunningJobsRegistry.java:98

	@Override
	public void clearJob(JobID jobID) throws IOException {
		checkNotNull(jobID);

		try {
			kubeClient.checkAndUpdateConfigMap(
				configMapName,
				configMap -> {
					if (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {
						if (configMap.getData().remove(getKeyForJobId(jobID)) != null) {
							return Optional.of(configMap);
						}
					}
					return Optional.empty();
				}
			).get();
		} catch (Exception e) {
			throw new IOException("Failed to clear job state in ConfigMap " + configMapName + " for job " + jobID, e);
		}
	}

	private void writeJobStatusToConfigMap(JobID jobID, JobSchedulingStatus status) throws IOException {
		LOG.debug("Setting scheduling state for job {} to {}.", jobID, status);
		final String key = getKeyForJobId(jobID);
		try {
			kubeClient.checkAndUpdateConfigMap(
				configMapName,
				configMap -> {
					if (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {
						final Optional<JobSchedulingStatus> optional = getJobStatus(configMap, jobID);
						if (!optional.isPresent() || optional.get() != status) {
							configMap.getData().put(key, status.name());
							return Optional.of(configMap);
						}
					}
					return Optional.empty();

View on GitHub (pinned to d731cee761)

Solutions

1. Check cluster/API-server connectivity (kubectl get configmaps in the HA namespace) and retry clearJob.
2. Verify the caller still holds leadership with the same lockIdentity; re-acquire leadership before clearing.
3. Confirm the ConfigMap named by the HA config (kubernetes.high-availability.config-map) exists and is not read-only or immutable.
4. Inspect the wrapped cause 'e' in the message/log for the root KubernetesException (409 conflict, 404, timeouts).

Example fix

// before: clear during transient API outage fails unrecoverably
runningJobsRegistry.clearJob(jobID);
// after: tolerate transient failures with retry
for (int i = 0; i < 3; i++) {
    try {
        runningJobsRegistry.clearJob(jobID);
        break;
    } catch (IOException e) {
        if (i == 2) throw e;
        Thread.sleep(1000);
    }
}

// before clearing, confirm prerequisites
KubernetesConfigMap cm = kubeClient.getConfigMap(haConfigMapName).orElseThrow(
    () -> new IllegalStateException("HA ConfigMap missing"));
boolean leader = KubernetesLeaderElector.hasLeadership(cm, lockIdentity);

try {
    registry.clearJob(jobID);
} catch (IOException e) {
    LOG.warn("Deferred: failed to clear state for {}", jobID, e); // retry later
}

When it happens

Trigger: Calling clearJob(jobID) when the target ConfigMap does not exist, the caller lost leadership (lock identity no longer matches), the Kubernetes API server is unreachable, or the update conflicts with a concurrent modification (resourceVersion conflict).

Common situations: Job termination during a Kubernetes API server outage or network partition; leader change mid-clear causing the hasLeadership check to fail; ConfigMap deleted by another component before the clear.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Failed to set {} state in ConfigMap {} for job {}
• {} already exists in ConfigMap {}
• Could not find {} in ConfigMap {}
• ConfigMap {configMapName} does not exists. It may be deleted
• Failed to update ConfigMap {} since current KubernetesCheckp

AI-assisted analysis of zhisheng17/flink-learning@d731cee761 (2026-09-06). Data as JSON: /api/errors/1d135f9abd64ce98. Report an issue: GitHub.