argoproj/argo-workflows · error

failed to delete ConfigMap %s: %w

Error message

failed to delete ConfigMap %s: %w

What it means

After pruning expired entries, if a memoization ConfigMap becomes empty the cache-GC deletes it outright; a Delete error other than NotFound is wrapped as 'failed to delete ConfigMap' and aborts GC for that ConfigMap. This keeps memoization ConfigMaps from accumulating as empty husks after all entries expire.

Source

Thrown at workflow/controller/cache_gc.go:60

	var modified bool
	for key, rawEntry := range cm.Data {
		var entry controllercache.Entry
		if err := json.Unmarshal([]byte(rawEntry), &entry); err != nil {
			return fmt.Errorf("malformed cache entry: could not unmarshal JSON; unable to parse: %w", err)
		}
		if time.Since(entry.LastHitTimestamp.Time) > wfc.gcAfterNotHitDuration {
			logger.WithFields(logging.Fields{"key": key, "configMap": cm.Name, "gcAfterNotHitDuration": wfc.gcAfterNotHitDuration}).Info(ctx, "Deleting entry in ConfigMap since it's not been hit")
			delete(cm.Data, key)
			modified = true
		}
	}
	if len(cm.Data) == 0 {
		err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Delete(ctx, cm.Name, metav1.DeleteOptions{})
		if err != nil {
			if apierr.IsNotFound(err) {
				return nil
			}
			return fmt.Errorf("failed to delete ConfigMap %s: %w", cm.Name, err)
		}
	} else if modified {
		_, err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Update(ctx, cm, metav1.UpdateOptions{})
		if err != nil {
			return fmt.Errorf("failed to update ConfigMap %s: %w", cm.Name, err)
		}
	}

	return nil
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped underlying error in controller logs to identify the exact Kubernetes cause (Forbidden / Conflict / webhook denial).
2. If Forbidden: grant the workflow-controller SA delete on configmaps in the memoization namespace.
3. If a finalizer or admission webhook blocks deletion, remove the blocking finalizer/webhook config or exclude memoization ConfigMaps from that webhook.
4. If the namespace is terminating, let it finish; GC will retry next sync cycle.
5. Verify with kubectl get cm <name> -o yaml that nothing (ownerReferences/finalizers) pins the ConfigMap.

Example fix

# before
verbs: ["get", "update"]
# after: allow GC to delete emptied cache ConfigMaps
verbs: ["create", "get", "update", "delete"]

kubectl auth can-i delete configmaps -n <ns> --as=system:serviceaccount:<argo-ns>:workflow-controller
kubectl get cm <name> -o jsonpath='{.metadata.finalizers}'  # must be empty for clean delete

err := wfc.kubeclientset.CoreV1().ConfigMaps(ns).Delete(ctx, name, metav1.DeleteOptions{})
if err != nil && !apierr.IsNotFound(err) {
  if apierr.IsForbidden(err) { /* grant delete on configmaps */ }
  if apierr.IsConflict(err) { /* terminating namespace or finalizer: wait and let GC retry */ }
  return err
}

When it happens

Trigger: wfc.kubeclientset delete on the ConfigMap fails with a non-NotFound error: RBAC forbids delete, finalizers block deletion, API server unreachable/admission webhook rejection, or namespace terminating (409).

Common situations: Controller SA missing the delete verb on configmaps; third-party finalizers or mutating webhooks intercepting ConfigMap deletes; ConfigMap stuck in a terminating namespace; air-gapped clusters with API instability during the GC sync.

Related errors

• could not save to config map cache: %w
• error creating cache entry: %w. Please check out this page f
• malformed cache entry: could not unmarshal JSON; unable to p
• failed to update ConfigMap %s: %w
• memoization configmap doesn't have %s label, refusing to use

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/4f377b51359489b2. Report an issue: GitHub.