{"record":{"id":"ff7fa2b56a62f631","repo":"zhisheng17/flink-learning","slug":"failed-to-set-state-in-configmap-for-job","errorCode":null,"errorMessage":"Failed to set {} state in ConfigMap {} for job {}","messagePattern":"Failed to set (.+?) state in ConfigMap (.+?) for job (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesRunningJobsRegistry.java","lineNumber":120,"sourceCode":"\tprivate void writeJobStatusToConfigMap(JobID jobID, JobSchedulingStatus status) throws IOException {\n\t\tLOG.debug(\"Setting scheduling state for job {} to {}.\", jobID, status);\n\t\tfinal String key = getKeyForJobId(jobID);\n\t\ttry {\n\t\t\tkubeClient.checkAndUpdateConfigMap(\n\t\t\t\tconfigMapName,\n\t\t\t\tconfigMap -> {\n\t\t\t\t\tif (KubernetesLeaderElector.hasLeadership(configMap, lockIdentity)) {\n\t\t\t\t\t\tfinal Optional<JobSchedulingStatus> optional = getJobStatus(configMap, jobID);\n\t\t\t\t\t\tif (!optional.isPresent() || optional.get() != status) {\n\t\t\t\t\t\t\tconfigMap.getData().put(key, status.name());\n\t\t\t\t\t\t\treturn Optional.of(configMap);\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\treturn Optional.empty();\n\t\t\t\t}\n\t\t\t).get();\n\t\t} catch (Exception e) {\n\t\t\tthrow new IOException(\"Failed to set \" + status.name() + \" state in ConfigMap \"\n\t\t\t\t+ configMapName + \" for job \" + jobID, e);\n\t\t}\n\t}\n\n\tprivate Optional<JobSchedulingStatus> getJobStatus(KubernetesConfigMap configMap, JobID jobId) {\n\t\tfinal String key = getKeyForJobId(jobId);\n\t\tfinal String status = configMap.getData().get(key);\n\t\tif (!StringUtils.isNullOrWhitespaceOnly(status)) {\n\t\t\treturn Optional.of(JobSchedulingStatus.valueOf(status));\n\t\t}\n\t\treturn Optional.empty();\n\t}\n\n\tprivate String getKeyForJobId(JobID jobId) {\n\t\treturn RUNNING_JOBS_REGISTRY_KEY_PREFIX + jobId.toString();\n\t}\n}\n","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/zhisheng17/flink-learning/blob/d731cee7618021be56d132cc925102ffff8d75e6/flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesRunningJobsRegistry.java#L102-L138","documentation":"writeJobStatusToConfigMap() wraps any failure from the ConfigMap read-modify-write that sets a job's status (Running or Finished) into this IOException. The key write is rejected or the Kubernetes update fails, so the HA registry cannot record the new scheduling state.","triggerScenarios":"setJobRunning(jobID) or setJobFinished(jobID) invoked when the ConfigMap is missing, the caller lost leadership (lock identity mismatch), the API server returns an error, or a resourceVersion conflict occurs because another leader wrote concurrently.","commonSituations":"Split-brain / stale leader trying to mark a job finished; Kubernetes API throttling during cluster-wide job restarts; ConfigMap made immutable; namespace RBAC forbidding ConfigMap updates.","solutions":["Verify leadership and lockIdentity are current; retry setJobRunning/setJobFinished after re-acquiring leadership.","Check RBAC: the Flink service account needs get/update on ConfigMaps in the HA namespace.","Check the wrapped cause for 409 conflicts and retry with backoff.","Confirm the HA ConfigMap exists and is not immutable (Immutable ConfigMaps reject updates)."],"exampleFix":"// before: update fails with immutable configmap\nkubectl patch cm flink-cluster-1 -p '{\"data\":{\"job-0001\":\"RUNNING\"}}'\n// after: remove immutable flag\nkubectl patch cm flink-cluster-1 --type merge -p '{\"immutable\": false}'","handlingStrategy":"retry","validationCode":"// verify write access before registering job state\nkubeClient.getConfigMap(haConfigMapName).orElseThrow(\n    () -> new IllegalStateException(\"HA ConfigMap does not exist\"));\n// and RBAC: kubectl auth can-i update configmaps -n <ns> --as=system:serviceaccount:<ns>:<flink-sa>","typeGuard":null,"tryCatchPattern":"try {\n    registry.setJobFinished(jobID);\n} catch (IOException e) {\n    if (ExceptionUtils.findThrowable(e, KubernetesException.class).isPresent()) {\n        // re-acquire leadership then retry\n    }\n}","preventionTips":["Grant the Flink service account configmap get/update RBAC.","Keep HA ConfigMaps mutable (immutable: false).","Retry state writes under fresh leadership.","Watch for 409 conflicts indicating concurrent leaders."],"tags":["kubernetes","high-availability","configmap","leader-election"],"backgroundTag":"api-request-failed","analyzedSha":"d731cee7618021be56d132cc925102ffff8d75e6","analyzedAt":"2026-09-06T05:35:08.496Z","contentChangedAt":"2026-09-06T05:35:08.496Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}