argoproj/argo-workflows · error

malformed cache entry: could not unmarshal JSON; unable to p

Error message

malformed cache entry: could not unmarshal JSON; unable to parse: %w

What it means

The cache GC controller scans every memoization ConfigMap, unmarshalling each data value into cache.Entry to compare LastHitTimestamp against gcAfterNotHitDuration. If any entry's raw string is not valid JSON matching the Entry schema, cleanupUnusedCache returns this wrapped parse error and aborts GC for that ConfigMap, so stale entries can never be collected until it is fixed.

Source

Thrown at workflow/controller/cache_gc.go:46

	for _, obj := range configMaps {
		cm, ok := obj.(*apiv1.ConfigMap)
		if !ok {
			logger.Error(ctx, "Unable to convert object to configmap when syncing ConfigMaps")
			continue
		}
		if err := wfc.cleanupUnusedCache(ctx, cm); err != nil {
			logger.WithField("configMap", cm.Name).WithError(err).Error(ctx, "Unable to sync ConfigMap")
		}
	}
}

func (wfc *WorkflowController) cleanupUnusedCache(ctx context.Context, cm *apiv1.ConfigMap) error {
	logger := logging.RequireLoggerFromContext(ctx)
	var modified bool
	for key, rawEntry := range cm.Data {
		var entry controllercache.Entry
		if err := json.Unmarshal([]byte(rawEntry), &entry); err != nil {
			return fmt.Errorf("malformed cache entry: could not unmarshal JSON; unable to parse: %w", err)
		}
		if time.Since(entry.LastHitTimestamp.Time) > wfc.gcAfterNotHitDuration {
			logger.WithFields(logging.Fields{"key": key, "configMap": cm.Name, "gcAfterNotHitDuration": wfc.gcAfterNotHitDuration}).Info(ctx, "Deleting entry in ConfigMap since it's not been hit")
			delete(cm.Data, key)
			modified = true
		}
	}
	if len(cm.Data) == 0 {
		err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Delete(ctx, cm.Name, metav1.DeleteOptions{})
		if err != nil {
			if apierr.IsNotFound(err) {
				return nil
			}
			return fmt.Errorf("failed to delete ConfigMap %s: %w", cm.Name, err)
		}
	} else if modified {
		_, err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Update(ctx, cm, metav1.UpdateOptions{})
		if err != nil {

View on GitHub (pinned to 35bff19146)

Solutions

1. kubectl get cm <name> -o yaml and find the data value(s) that fail json.Unmarshal against cache.Entry fields.
2. Delete the corrupted key (kubectl patch/cm edit) so GC resumes for remaining valid entries.
3. If the whole ConfigMap is foreign or stale-schema, delete the entire ConfigMap — GC will not manage it and memoization entries rebuild on demand.
4. Stop external writers: ensure only the Argo controller writes to ConfigMaps carrying the memoization labels.
5. After cleanup, watch controller logs — GC re-scans on its sync period and will proceed once all entries parse.

Example fix

# before: corrupted key blocks GC
# after: remove just the bad key
kubectl patch cm <cache-cm> --type merge -p '{"data":{"<bad-key>":null}}'

const cm = await k8s.readConfigMap(ns, name);
for (const [k, v] of Object.entries(cm.data ?? {})) {
  try { const e = JSON.parse(v); if (!('nodeID' in e) && !('exitCode' in e)) throw 0; }
  catch { console.warn(`ConfigMap ${name}: entry '${k}' is not a valid cache Entry — GC will stall`); }
}

function isCacheEntry(raw) {
  if (typeof raw !== 'string') return false;
  try { const e = JSON.parse(raw); return e && typeof e === 'object' && 'nodeID' in e; } catch { return false; }
}

if err := wfc.cleanupUnusedCache(ctx, cm); err != nil {
  if strings.Contains(err.Error(), "could not unmarshal JSON") {
    // quarantine: delete the offending key(s), let the next GC sync proceed
  }
}

When it happens

Trigger: A memoization ConfigMap contains a data value that is not valid Entry JSON — written manually, written by another tool, truncated, or created by an incompatible Argo version's schema.

Common situations: Operators hand-editing memoization ConfigMaps; leftover entries from an upgraded Argo version with changed Entry fields; external tooling reusing the same ConfigMap (selector-matched by workflows.argoproj.io labels) with arbitrary values.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• malformed cache entry: could not unmarshal JSON; unable to p
• failed to delete ConfigMap %s: %w
• unable to marshal cache entry with last hit timestamp: %w
• could not save to config map cache: %w
• unable to marshal cache entry: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/c3deba978193b437. Report an issue: GitHub.