{"record":{"id":"1d135f9abd64ce98","repo":"zhisheng17/flink-learning","slug":"failed-to-clear-job-state-in-configmap-for-job","errorCode":null,"errorMessage":"Failed to clear job state in ConfigMap {} for job {}","messagePattern":"Failed to clear job state in ConfigMap (.+?) for job (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesRunningJobsRegistry.java","lineNumber":98,"sourceCode":"\n\t@Override\n\tpublic void clearJob(JobID jobID) throws IOException {\n\t\tcheckNotNull(jobID);\n\n\t\ttry {\n\t\t\tkubeClient.checkAndUpdateConfigMap(\n\t\t\t\tconfigMapName,\n\t\t\t\tconfigMap -> {\n\t\t\t\t\tif (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {\n\t\t\t\t\t\tif (configMap.getData().remove(getKeyForJobId(jobID)) != null) {\n\t\t\t\t\t\t\treturn Optional.of(configMap);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\treturn Optional.empty();\n\t\t\t\t}\n\t\t\t).get();\n\t\t} catch (Exception e) {\n\t\t\tthrow new IOException(\"Failed to clear job state in ConfigMap \" + configMapName + \" for job \" + jobID, e);\n\t\t}\n\t}\n\n\tprivate void writeJobStatusToConfigMap(JobID jobID, JobSchedulingStatus status) throws IOException {\n\t\tLOG.debug(\"Setting scheduling state for job {} to {}.\", jobID, status);\n\t\tfinal String key = getKeyForJobId(jobID);\n\t\ttry {\n\t\t\tkubeClient.checkAndUpdateConfigMap(\n\t\t\t\tconfigMapName,\n\t\t\t\tconfigMap -> {\n\t\t\t\t\tif (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {\n\t\t\t\t\t\tfinal Optional<JobSchedulingStatus> optional = getJobStatus(configMap, jobID);\n\t\t\t\t\t\tif (!optional.isPresent() || optional.get() != status) {\n\t\t\t\t\t\t\tconfigMap.getData().put(key, status.name());\n\t\t\t\t\t\t\treturn Optional.of(configMap);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\treturn Optional.empty();","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/zhisheng17/flink-learning/blob/d731cee7618021be56d132cc925102ffff8d75e6/flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesRunningJobsRegistry.java#L80-L116","documentation":"KubernetesRunningJobsRegistry.clearJob() wraps any exception from its read-modify-write ConfigMap transaction (removing the job's status key) into this IOException. The Kubernetes API call inside the lambda or the blocking .get() failed, so the job's scheduling state could not be cleared from the HA ConfigMap.","triggerScenarios":"Calling clearJob(jobID) when the target ConfigMap does not exist, the caller lost leadership (lock identity no longer matches), the Kubernetes API server is unreachable, or the update conflicts with a concurrent modification (resourceVersion conflict).","commonSituations":"Job termination during a Kubernetes API server outage or network partition; leader change mid-clear causing the hasLeadership check to fail; ConfigMap deleted by another component before the clear.","solutions":["Check cluster/API-server connectivity (kubectl get configmaps in the HA namespace) and retry clearJob.","Verify the caller still holds leadership with the same lockIdentity; re-acquire leadership before clearing.","Confirm the ConfigMap named by the HA config (kubernetes.high-availability.config-map) exists and is not read-only or immutable.","Inspect the wrapped cause 'e' in the message/log for the root KubernetesException (409 conflict, 404, timeouts)."],"exampleFix":"// before: clear during transient API outage fails unrecoverably\nrunningJobsRegistry.clearJob(jobID);\n// after: tolerate transient failures with retry\nfor (int i = 0; i < 3; i++) {\n    try {\n        runningJobsRegistry.clearJob(jobID);\n        break;\n    } catch (IOException e) {\n        if (i == 2) throw e;\n        Thread.sleep(1000);\n    }\n}","handlingStrategy":"try-catch","validationCode":"// before clearing, confirm prerequisites\nKubernetesConfigMap cm = kubeClient.getConfigMap(haConfigMapName).orElseThrow(\n    () -> new IllegalStateException(\"HA ConfigMap missing\"));\nboolean leader = KubernetesLeaderElector.hasLeadership(cm, lockIdentity);","typeGuard":null,"tryCatchPattern":"try {\n    registry.clearJob(jobID);\n} catch (IOException e) {\n    LOG.warn(\"Deferred: failed to clear state for {}\", jobID, e); // retry later\n}","preventionTips":["Retry clearJob with backoff; clearing job state is idempotent.","Only clear while holding confirmed leadership.","Monitor API-server health before job teardown operations.","Never mark the ConfigMap immutable."],"tags":["kubernetes","high-availability","configmap","io"],"backgroundTag":"api-request-failed","analyzedSha":"d731cee7618021be56d132cc925102ffff8d75e6","analyzedAt":"2026-09-06T05:35:08.496Z","contentChangedAt":"2026-09-06T05:35:08.496Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}