What causes "unable to marshal cache entry with last hit timestamp: %w" in argoproj/argo-workflows?

Retry the memoization lookup — transient; a fresh entry will be re-created on re-execution.

unable to marshal cache entry with last hit timestamp: %w

Error message

unable to marshal cache entry with last hit timestamp: %w

What it means

After a cache hit, load() updates the entry's LastHitTimestamp and re-marshals the Entry back to JSON before writing it to the ConfigMap. json.Marshal of a populated Entry essentially never fails in practice; if it does, the load is aborted with this wrapped error rather than returning a corrupted cache entry.

Source

Thrown at workflow/controller/cache/configmap_cache.go:124

	c.logInfo(ctx, logging.Fields{}, "config map cache loaded")
	hitTime := time.Now()
	rawEntry, ok := cm.Data[key]
	if !ok || rawEntry == "" {
		c.logInfo(ctx, logging.Fields{}, "config map cache miss: entry does not exist")
		return nil, nil
	}

	var entry Entry
	err = json.Unmarshal([]byte(rawEntry), &entry)
	if err != nil {
		return nil, fmt.Errorf("malformed cache entry: could not unmarshal JSON; unable to parse: %w", err)
	}

	entry.LastHitTimestamp = metav1.Time{Time: hitTime}
	entryJSON, err := json.Marshal(entry)
	if err != nil {
		c.logError(ctx, err, logging.Fields{"key": key}, "Unable to marshal cache entry with last hit timestamp")
		return nil, fmt.Errorf("unable to marshal cache entry with last hit timestamp: %w", err)
	}
	cm.Data[key] = string(entryJSON)

	_, err = c.kubeClient.CoreV1().ConfigMaps(c.namespace).Update(ctx, cm, metav1.UpdateOptions{})
	if err != nil {
		return nil, err
	}
	return &entry, nil
}

func (c *configMapCache) Save(ctx context.Context, key string, nodeID string, value *wfv1.Outputs) error {
	err := retry.OnError(kwait.Backoff{
		Duration: time.Second,
		Factor:   2,
		Jitter:   0.1,
		Steps:    5,
		Cap:      30 * time.Second,
	}, func(err error) bool {

View on GitHub (pinned to 35bff19146)

Solutions

Retry the memoization lookup — transient; a fresh entry will be re-created on re-execution.
Delete the offending ConfigMap key so the node re-executes and saves a clean entry.
If running a patched/forked controller, review recent changes to cache.Entry for fields that json.Marshal cannot serialize (channels, funcs, cycles).
Upgrade to a released Argo version — stock Entry (all strings/timestamps) always marshals.

Defensive patterns

Strategy: retry

Try / catch

entry, err := cache.Load(ctx, key)
if err != nil {
  if strings.Contains(err.Error(), "unable to marshal cache entry") {
    // transient/serialization issue: treat as miss and retry once
    entry, err = cache.Load(ctx, key)
  }
  if err != nil { return err }
}

Prevention

Keep cache.Entry JSON-serializable if forking/extending the controller.
On repeated failures, delete the ConfigMap key and let memoization rebuild the entry.
Report persistent occurrences upstream — stock Entry cannot trigger this.
Treat as a cache miss rather than a hard failure in wrappers around Load.

When it happens

Trigger: json.Marshal(entry) returning an error after setting LastHitTimestamp — theoretically only via unsupported field states (e.g. a corrupted time value in the entry being re-serialized).

Common situations: Practically unreachable for well-formed entries; seen only when Entry struct gains a field whose serialization fails or when upstream code injects an Entry with unserializable data in a fork/patch.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/ef4b66c6c3379d1a. Report an issue: GitHub.