argoproj/argo-workflows · warning

failed to update ConfigMap %s: %w

Error message

failed to update ConfigMap %s: %w

What it means

Returned by cleanupUnusedCache in workflow/controller/cache_gc.go:65 when the periodic memoization-cache GC removes stale entries from a cache ConfigMap and the subsequent ConfigMap Update API call fails. The underlying Kubernetes client error is wrapped with %w, so errors.Is/As on the cause (e.g. Conflict, Forbidden) works. It is logged (not fatal) by syncAllCacheForGC and retried on the next GC tick.

Source

Thrown at workflow/controller/cache_gc.go:65

		}
		if time.Since(entry.LastHitTimestamp.Time) > wfc.gcAfterNotHitDuration {
			logger.WithFields(logging.Fields{"key": key, "configMap": cm.Name, "gcAfterNotHitDuration": wfc.gcAfterNotHitDuration}).Info(ctx, "Deleting entry in ConfigMap since it's not been hit")
			delete(cm.Data, key)
			modified = true
		}
	}
	if len(cm.Data) == 0 {
		err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Delete(ctx, cm.Name, metav1.DeleteOptions{})
		if err != nil {
			if apierr.IsNotFound(err) {
				return nil
			}
			return fmt.Errorf("failed to delete ConfigMap %s: %w", cm.Name, err)
		}
	} else if modified {
		_, err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Update(ctx, cm, metav1.UpdateOptions{})
		if err != nil {
			return fmt.Errorf("failed to update ConfigMap %s: %w", cm.Name, err)
		}
	}

	return nil
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Retry is built in: the next GC cycle re-reads the ConfigMap from the informer and re-applies deletion — usually no action needed.
2. If conflicts are frequent, reduce concurrent writers to the cache ConfigMap (fewer memoization hits racing) or accept GC lag.
3. Check RBAC: the controller service account needs configmaps update/delete on the workflow namespace.
4. Verify the ConfigMap still exists and its size is under the 1MiB limit; prune huge cache entries or use a shorter gcAfterNotHitDuration.

// Before relying on GC, confirm RBAC and concurrency
const cm = await k8s.readNamespacedConfigMap(name, ns)
// ensure resourceVersion is fresh before update; expect 409 under contention

try {
  await gcCacheConfigMap(cm)
} catch (err) {
  if (isConflict(err)) { /* resync from informer; retry next GC tick */ }
  else if (isForbidden(err)) { fixControllerRBAC() }
  else { logAndRetryLater(err) }
}

When it happens

Trigger: GC deletes expired entries from a cache ConfigMap (workflow-controller-configmap referenced by a memoize template with caches->configMap) and calls CoreV1().ConfigMaps(...).Update(ctx, cm, ...) which fails — most commonly an optimistic-concurrency 409 Conflict because the memoization loop or another controller updated the ConfigMap since it was read from the informer, or RBAC failure, or the ConfigMap was deleted concurrently (404).

Common situations: Heavy memoization traffic on the same cache ConfigMap causing constant update conflicts; controller service account lacking update permission on ConfigMaps in the namespace; the cache ConfigMap deleted mid-GC; extremely large ConfigMaps nearing the 1MiB limit so writes are rejected.

Related errors

• failed to delete ConfigMap %s: %w
• memoization configmap doesn't have %s label, refusing to use
• failed to unmarshal container args: %w
• AlreadyExists
• error getting config map for artifact repository ref "%v": %

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/fb4a656ac7de20af. Report an issue: GitHub.