{"record":{"id":"c3deba978193b437","repo":"argoproj/argo-workflows","slug":"malformed-cache-entry-could-not-unmarshal-json-u-c3deba","errorCode":null,"errorMessage":"malformed cache entry: could not unmarshal JSON; unable to parse: %w","messagePattern":"malformed cache entry: could not unmarshal JSON; unable to parse: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/controller/cache_gc.go","lineNumber":46,"sourceCode":"\tfor _, obj := range configMaps {\n\t\tcm, ok := obj.(*apiv1.ConfigMap)\n\t\tif !ok {\n\t\t\tlogger.Error(ctx, \"Unable to convert object to configmap when syncing ConfigMaps\")\n\t\t\tcontinue\n\t\t}\n\t\tif err := wfc.cleanupUnusedCache(ctx, cm); err != nil {\n\t\t\tlogger.WithField(\"configMap\", cm.Name).WithError(err).Error(ctx, \"Unable to sync ConfigMap\")\n\t\t}\n\t}\n}\n\nfunc (wfc *WorkflowController) cleanupUnusedCache(ctx context.Context, cm *apiv1.ConfigMap) error {\n\tlogger := logging.RequireLoggerFromContext(ctx)\n\tvar modified bool\n\tfor key, rawEntry := range cm.Data {\n\t\tvar entry controllercache.Entry\n\t\tif err := json.Unmarshal([]byte(rawEntry), &entry); err != nil {\n\t\t\treturn fmt.Errorf(\"malformed cache entry: could not unmarshal JSON; unable to parse: %w\", err)\n\t\t}\n\t\tif time.Since(entry.LastHitTimestamp.Time) > wfc.gcAfterNotHitDuration {\n\t\t\tlogger.WithFields(logging.Fields{\"key\": key, \"configMap\": cm.Name, \"gcAfterNotHitDuration\": wfc.gcAfterNotHitDuration}).Info(ctx, \"Deleting entry in ConfigMap since it's not been hit\")\n\t\t\tdelete(cm.Data, key)\n\t\t\tmodified = true\n\t\t}\n\t}\n\tif len(cm.Data) == 0 {\n\t\terr := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Delete(ctx, cm.Name, metav1.DeleteOptions{})\n\t\tif err != nil {\n\t\t\tif apierr.IsNotFound(err) {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\treturn fmt.Errorf(\"failed to delete ConfigMap %s: %w\", cm.Name, err)\n\t\t}\n\t} else if modified {\n\t\t_, err := wfc.kubeclientset.CoreV1().ConfigMaps(cm.Namespace).Update(ctx, cm, metav1.UpdateOptions{})\n\t\tif err != nil {","sourceCodeStart":28,"sourceCodeEnd":64,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/cache_gc.go#L28-L64","documentation":"The cache GC controller scans every memoization ConfigMap, unmarshalling each data value into cache.Entry to compare LastHitTimestamp against gcAfterNotHitDuration. If any entry's raw string is not valid JSON matching the Entry schema, cleanupUnusedCache returns this wrapped parse error and aborts GC for that ConfigMap, so stale entries can never be collected until it is fixed.","triggerScenarios":"A memoization ConfigMap contains a data value that is not valid Entry JSON — written manually, written by another tool, truncated, or created by an incompatible Argo version's schema.","commonSituations":"Operators hand-editing memoization ConfigMaps; leftover entries from an upgraded Argo version with changed Entry fields; external tooling reusing the same ConfigMap (selector-matched by workflows.argoproj.io labels) with arbitrary values.","solutions":["kubectl get cm <name> -o yaml and find the data value(s) that fail json.Unmarshal against cache.Entry fields.","Delete the corrupted key (kubectl patch/cm edit) so GC resumes for remaining valid entries.","If the whole ConfigMap is foreign or stale-schema, delete the entire ConfigMap — GC will not manage it and memoization entries rebuild on demand.","Stop external writers: ensure only the Argo controller writes to ConfigMaps carrying the memoization labels.","After cleanup, watch controller logs — GC re-scans on its sync period and will proceed once all entries parse."],"exampleFix":"# before: corrupted key blocks GC\n# after: remove just the bad key\nkubectl patch cm <cache-cm> --type merge -p '{\"data\":{\"<bad-key>\":null}}'","handlingStrategy":"validation","validationCode":"const cm = await k8s.readConfigMap(ns, name);\nfor (const [k, v] of Object.entries(cm.data ?? {})) {\n  try { const e = JSON.parse(v); if (!('nodeID' in e) && !('exitCode' in e)) throw 0; }\n  catch { console.warn(`ConfigMap ${name}: entry '${k}' is not a valid cache Entry — GC will stall`); }\n}","typeGuard":"function isCacheEntry(raw) {\n  if (typeof raw !== 'string') return false;\n  try { const e = JSON.parse(raw); return e && typeof e === 'object' && 'nodeID' in e; } catch { return false; }\n}","tryCatchPattern":"if err := wfc.cleanupUnusedCache(ctx, cm); err != nil {\n  if strings.Contains(err.Error(), \"could not unmarshal JSON\") {\n    // quarantine: delete the offending key(s), let the next GC sync proceed\n  }\n}","preventionTips":["Audit memoization ConfigMaps after Argo upgrades for schema changes.","Block external tools from writing to ConfigMaps carrying memoization labels.","Delete entire foreign/stale-schema ConfigMaps rather than letting GC stall on them.","Alert on this controller error — one bad key blocks GC of all entries in that ConfigMap."],"tags":["memoization","cache","gc","json"],"backgroundTag":"malformed-cache-entry","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}