{"record":{"id":"4f377b51359489b2","repo":"argoproj/argo-workflows","slug":"failed-to-delete-configmap-s-w","errorCode":null,"errorMessage":"failed to delete ConfigMap %s: %w","messagePattern":"failed to delete ConfigMap (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/controller/cache_gc.go","lineNumber":60,"sourceCode":"\tvar modified bool\n\tfor key, rawEntry := range cm.Data {\n\t\tvar entry controllercache.Entry\n\t\tif err := json.Unmarshal([]byte(rawEntry), &entry); err != nil {\n\t\t\treturn fmt.Errorf(\"malformed cache entry: could not unmarshal JSON; unable to parse: %w\", err)\n\t\t}\n\t\tif time.Since(entry.LastHitTimestamp.Time) > wfc.gcAfterNotHitDuration {\n\t\t\tlogger.WithFields(logging.Fields{\"key\": key, \"configMap\": cm.Name, \"gcAfterNotHitDuration\": wfc.gcAfterNotHitDuration}).Info(ctx, \"Deleting entry in ConfigMap since it's not been hit\")\n\t\t\tdelete(cm.Data, key)\n\t\t\tmodified = true\n\t\t}\n\t}\n\tif len(cm.Data) == 0 {\n\t\terr := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Delete(ctx, cm.Name, metav1.DeleteOptions{})\n\t\tif err != nil {\n\t\t\tif apierr.IsNotFound(err) {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\treturn fmt.Errorf(\"failed to delete ConfigMap %s: %w\", cm.Name, err)\n\t\t}\n\t} else if modified {\n\t\t_, err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Update(ctx, cm, metav1.UpdateOptions{})\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to update ConfigMap %s: %w\", cm.Name, err)\n\t\t}\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":42,"sourceCodeEnd":71,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/cache_gc.go#L42-L71","documentation":"After pruning expired entries, if a memoization ConfigMap becomes empty the cache-GC deletes it outright; a Delete error other than NotFound is wrapped as 'failed to delete ConfigMap' and aborts GC for that ConfigMap. This keeps memoization ConfigMaps from accumulating as empty husks after all entries expire.","triggerScenarios":"wfc.kubeclientset delete on the ConfigMap fails with a non-NotFound error: RBAC forbids delete, finalizers block deletion, API server unreachable/admission webhook rejection, or namespace terminating (409).","commonSituations":"Controller SA missing the delete verb on configmaps; third-party finalizers or mutating webhooks intercepting ConfigMap deletes; ConfigMap stuck in a terminating namespace; air-gapped clusters with API instability during the GC sync.","solutions":["Read the wrapped underlying error in controller logs to identify the exact Kubernetes cause (Forbidden / Conflict / webhook denial).","If Forbidden: grant the workflow-controller SA delete on configmaps in the memoization namespace.","If a finalizer or admission webhook blocks deletion, remove the blocking finalizer/webhook config or exclude memoization ConfigMaps from that webhook.","If the namespace is terminating, let it finish; GC will retry next sync cycle.","Verify with kubectl get cm <name> -o yaml that nothing (ownerReferences/finalizers) pins the ConfigMap."],"exampleFix":"# before\nverbs: [\"get\", \"update\"]\n# after: allow GC to delete emptied cache ConfigMaps\nverbs: [\"create\", \"get\", \"update\", \"delete\"]","handlingStrategy":"try-catch","validationCode":"kubectl auth can-i delete configmaps -n <ns> --as=system:serviceaccount:<argo-ns>:workflow-controller\nkubectl get cm <name> -o jsonpath='{.metadata.finalizers}'  # must be empty for clean delete","typeGuard":null,"tryCatchPattern":"err := wfc.kubeclientset.CoreV1().ConfigMaps(ns).Delete(ctx, name, metav1.DeleteOptions{})\nif err != nil && !apierr.IsNotFound(err) {\n  if apierr.IsForbidden(err) { /* grant delete on configmaps */ }\n  if apierr.IsConflict(err) { /* terminating namespace or finalizer: wait and let GC retry */ }\n  return err\n}","preventionTips":["Include the delete verb on configmaps in controller RBAC for all workflow namespaces.","Remove/avoid finalizers and mutating webhooks on memoization ConfigMaps.","Check for terminating namespaces when GC errors cluster in one namespace.","Monitor GC sync logs; delete is retried every cycle, so transient API errors self-heal."],"tags":["memoization","cache","gc","kubernetes"],"backgroundTag":"configmap-delete-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}