What causes "Could not properly remove all state handles." in zhisheng17/flink-learning?

Inspect the suppressed exceptions in the cause chain — usually only some handles failed; the ConfigMap keys were still removed.

zhisheng17/flink-learning · error · KubernetesException

Could not properly remove all state handles.

Error message

Could not properly remove all state handles.

What it means

releaseAndTryRemoveAll() collects discardState() failures for every state handle it removes and, if any occurred, wraps them in this CompletionException(KubernetesException). The keys were removed from the ConfigMap but at least one deserialized state handle could not be discarded (cleanup of external resources failed).

Source

Thrown at flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesStateHandleStore.java:377

						});
					c.getData().clear();
					c.getData().putAll(updateData);
					return Optional.of(c);
				}
				return Optional.empty();
			})
		.whenComplete((succeed, ignore) -> {
			if (succeed) {
				Exception exception = null;
				for (RetrievableStateHandle<T> stateHandle : validStateHandles) {
					try {
						stateHandle.discardState();
					} catch (Exception e) {
						exception = ExceptionUtils.firstOrSuppressed(e, exception);
					}
				}
				if (exception != null) {
					throw new CompletionException(new KubernetesException(
						"Could not properly remove all state handles.", exception));
				}
			}
		}).get();
	}

	/**
	 * Remove all the filtered keys in the ConfigMap.
	 *
	 * @throws Exception when removing the keys failed
	 */
	@Override
	public void clearEntries() throws Exception {
		kubeClient.checkAndUpdateConfigMap(
			configMapName,
			c -> {
				if (KubernetesLeaderElector.hasLeadership(c, lockIdentity)) {
					c.getData().keySet().removeIf(configMapKeyFilter);

View on GitHub (pinned to d731cee761)

Solutions

Inspect the suppressed exceptions in the cause chain — usually only some handles failed; the ConfigMap keys were still removed.
If failures are 'resource not found' during discard, they are benign for teardown and can be ignored/logged.
Restore access to the state backend (credentials, filesystem) and retry releaseAndTryRemoveAll.
For lingering handles, manually clean the HA ConfigMap keys after confirming nothing else depends on them.

Example fix

// before: teardown aborts on discard failure
store.releaseAndTryRemoveAll();
// after: tolerate best-effort cleanup
try {
    store.releaseAndTryRemoveAll();
} catch (Exception e) {
    LOG.warn("Best-effort cleanup had discard failures", e);
}

Defensive patterns

Strategy: try-catch

Validate before calling

// confirm backend reachability before discarding handles
boolean backendReachable = checkpointStorageFs.exists(baseDir); // cheap probe

Try / catch

try {
    stateHandleStore.releaseAndTryRemoveAll();
} catch (Exception e) {
    LOG.warn("Cleanup incomplete; ConfigMap keys removed, some discardState failed", e);
}

Prevention

Treat discard failures during teardown as best-effort.
Keep state backend credentials valid until after teardown.
Drain old state handles before Flink/jar version upgrades.
Log suppressed causes to identify which handles failed.

When it happens

Trigger: releaseAndTryRemoveAll() invoked when one or more stored state handles throw from discardState() — e.g. corrupted/deserialization-degraded handles, or the backing resource (checkpoint storage) is already gone.

Common situations: Cluster teardown after a storage backend (HDFS/S3) was decommissioned or credentials revoked; handles whose target files were manually deleted; mixing incompatible state-handle versions after a Flink upgrade.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of zhisheng17/flink-learning@d731cee761 (2026-09-06). Data as JSON: /api/errors/d56174414f20c4d4. Report an issue: GitHub.