{"record":{"id":"d56174414f20c4d4","repo":"zhisheng17/flink-learning","slug":"could-not-properly-remove-all-state-handles","errorCode":null,"errorMessage":"Could not properly remove all state handles.","messagePattern":"Could not properly remove all state handles\\.","errorType":"exception","errorClass":"KubernetesException","httpStatus":null,"severity":"error","filePath":"flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesStateHandleStore.java","lineNumber":377,"sourceCode":"\t\t\t\t\t\t});\n\t\t\t\t\tc.getData().clear();\n\t\t\t\t\tc.getData().putAll(updateData);\n\t\t\t\t\treturn Optional.of(c);\n\t\t\t\t}\n\t\t\t\treturn Optional.empty();\n\t\t\t})\n\t\t.whenComplete((succeed, ignore) -> {\n\t\t\tif (succeed) {\n\t\t\t\tException exception = null;\n\t\t\t\tfor (RetrievableStateHandle<T> stateHandle : validStateHandles) {\n\t\t\t\t\ttry {\n\t\t\t\t\t\tstateHandle.discardState();\n\t\t\t\t\t} catch (Exception e) {\n\t\t\t\t\t\texception = ExceptionUtils.firstOrSuppressed(e, exception);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tif (exception != null) {\n\t\t\t\t\tthrow new CompletionException(new KubernetesException(\n\t\t\t\t\t\t\"Could not properly remove all state handles.\", exception));\n\t\t\t\t}\n\t\t\t}\n\t\t}).get();\n\t}\n\n\t/**\n\t * Remove all the filtered keys in the ConfigMap.\n\t *\n\t * @throws Exception when removing the keys failed\n\t */\n\t@Override\n\tpublic void clearEntries() throws Exception {\n\t\tkubeClient.checkAndUpdateConfigMap(\n\t\t\tconfigMapName,\n\t\t\tc -> {\n\t\t\t\tif (KubernetesLeaderElector.hasLeadership(c, lockIdentity)) {\n\t\t\t\t\tc.getData().keySet().removeIf(configMapKeyFilter);","sourceCodeStart":359,"sourceCodeEnd":395,"githubUrl":"https://github.com/zhisheng17/flink-learning/blob/d731cee7618021be56d132cc925102ffff8d75e6/flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesStateHandleStore.java#L359-L395","documentation":"releaseAndTryRemoveAll() collects discardState() failures for every state handle it removes and, if any occurred, wraps them in this CompletionException(KubernetesException). The keys were removed from the ConfigMap but at least one deserialized state handle could not be discarded (cleanup of external resources failed).","triggerScenarios":"releaseAndTryRemoveAll() invoked when one or more stored state handles throw from discardState() — e.g. corrupted/deserialization-degraded handles, or the backing resource (checkpoint storage) is already gone.","commonSituations":"Cluster teardown after a storage backend (HDFS/S3) was decommissioned or credentials revoked; handles whose target files were manually deleted; mixing incompatible state-handle versions after a Flink upgrade.","solutions":["Inspect the suppressed exceptions in the cause chain — usually only some handles failed; the ConfigMap keys were still removed.","If failures are 'resource not found' during discard, they are benign for teardown and can be ignored/logged.","Restore access to the state backend (credentials, filesystem) and retry releaseAndTryRemoveAll.","For lingering handles, manually clean the HA ConfigMap keys after confirming nothing else depends on them."],"exampleFix":"// before: teardown aborts on discard failure\nstore.releaseAndTryRemoveAll();\n// after: tolerate best-effort cleanup\ntry {\n    store.releaseAndTryRemoveAll();\n} catch (Exception e) {\n    LOG.warn(\"Best-effort cleanup had discard failures\", e);\n}","handlingStrategy":"try-catch","validationCode":"// confirm backend reachability before discarding handles\nboolean backendReachable = checkpointStorageFs.exists(baseDir); // cheap probe","typeGuard":null,"tryCatchPattern":"try {\n    stateHandleStore.releaseAndTryRemoveAll();\n} catch (Exception e) {\n    LOG.warn(\"Cleanup incomplete; ConfigMap keys removed, some discardState failed\", e);\n}","preventionTips":["Treat discard failures during teardown as best-effort.","Keep state backend credentials valid until after teardown.","Drain old state handles before Flink/jar version upgrades.","Log suppressed causes to identify which handles failed."],"tags":["kubernetes","state-handle","cleanup","high-availability"],"backgroundTag":"resource-not-found","analyzedSha":"d731cee7618021be56d132cc925102ffff8d75e6","analyzedAt":"2026-09-06T05:35:08.496Z","contentChangedAt":"2026-09-06T05:35:08.496Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}