What causes "Could not find {} in ConfigMap {}" in zhisheng17/flink-learning?

Use addAndLock instead of replace when the key may not exist yet.

zhisheng17/flink-learning · error · NotExistException

Could not find {} in ConfigMap {}

Error message

Could not find {} in ConfigMap {}

What it means

KubernetesStateHandleStore.replace() throws this when the key to replace does not exist in the ConfigMap ( getKeyNotExistException(key) ), mapped via a CompletionException. replace() only updates existing keys — it is not an upsert.

Source

Thrown at flink-learning-k8s/flink-k8s/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesStateHandleStore.java:179

		checkNotNull(state, "State.");

		final RetrievableStateHandle<T> oldStateHandle = getAndLock(key);

		final RetrievableStateHandle<T> newStateHandle = storage.store(state);

		boolean success = false;

		try {
			final byte[] serializedStoreHandle = InstantiationUtil.serializeObject(newStateHandle);
			success = kubeClient.checkAndUpdateConfigMap(
				configMapName,
				c -> {
					if (KubernetesLeaderElector.hasLeadership(c, lockIdentity)) {
						// Check the existence
						if (c.getData().containsKey(key)) {
							c.getData().put(key, encodeStateHandle(serializedStoreHandle));
						} else {
							throw new CompletionException(getKeyNotExistException(key));
						}
						return Optional.of(c);
					}
					return Optional.empty();
				}).get();
		} catch (Exception ex) {
			throw ExceptionUtils.findThrowable(ex, NotExistException.class).orElseThrow(() -> ex);
		} finally {
			if (success) {
				oldStateHandle.discardState();
			} else {
				newStateHandle.discardState();
			}
		}
	}

	/**
	 * Returns the resource version of the ConfigMap.

View on GitHub (pinned to d731cee761)

Solutions

Use addAndLock instead of replace when the key may not exist yet.
Check existence first (check(key)) and branch between add and replace.
If replace races with removal, re-check leadership/job state — the key disappearing may be legitimate cleanup.
Verify the ConfigMap was not recreated (e.g. by re-deploying the HA setup) losing prior keys.

Example fix

// before: replace on possibly-missing key
stateHandleStore.replace(key, newHandle);
// after: add-or-replace
if (stateHandleStore.check(key).isPresent()) {
    stateHandleStore.replace(key, newHandle);
} else {
    stateHandleStore.addAndLock(key, newHandle);
}

Defensive patterns

Strategy: validation

Validate before calling

// verify key exists before replace
if (!stateHandleStore.check(key).isPresent()) {
    throw new IllegalStateException("Cannot replace missing key " + key);
}
stateHandleStore.replace(key, handle);

Try / catch

try {
    stateHandleStore.replace(key, handle);
} catch (Exception e) {
    if (e.getCause() != null && e.getCause().getMessage().contains("Could not find")) {
        stateHandleStore.addAndLock(key, handle); // add-or-replace fallback
    }
}

Prevention

Prefer addAndLock for first writes; replace only for updates.
Serialize replace/remove for the same key under one leader.
Recreate HA ConfigMaps carefully — keys are lost on recreation.
Use check(key) before destructive flows.

When it happens

Trigger: replace(key, stateHandle) called while holding leadership but the key is absent from the ConfigMap — e.g. the state handle was already removed, the ConfigMap was recreated empty, or replace raced with a concurrent remove.

Common situations: Updating a checkpoint/coordinator state handle after the job was cleaned up; two nodes racing where one removes the key before the other's replace; using replace on a fresh ConfigMap that only addAndLock may seed.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of zhisheng17/flink-learning@d731cee761 (2026-09-06). Data as JSON: /api/errors/641859fdb072c591. Report an issue: GitHub.