argoproj/argo-workflows · warning

failed to get workflow by key after locking

Error message

failed to get workflow by key after locking

What it means

deleteOffloadedNodesForWorkflow (workflow/controller/controller.go:849) locks the workflow key, then calls getWorkflowByKey; if the key is no longer present in the informer cache the error is returned. This is a benign race: between the initial UID-index lookup and acquiring the per-key lock, the workflow was deleted (or the cache re-synced), so the object vanished.

Source

Thrown at workflow/controller/controller.go:849

		return err
	}
	var wf *wfv1.Workflow
	logger := logging.RequireLoggerFromContext(ctx)
	switch l := len(workflows); l {
	case 0:
		logger.WithField("uid", uid).Info(ctx, "Workflow missing, probably deleted")
	case 1:
		un, ok := workflows[0].(*unstructured.Unstructured)
		if !ok {
			return fmt.Errorf("object %+v is not an unstructured", workflows[0])
		}
		key := un.GetNamespace() + "/" + un.GetName()
		wfc.workflowKeyLock.Lock(key)
		defer wfc.workflowKeyLock.Unlock(key)

		obj, ok := wfc.getWorkflowByKey(ctx, key)
		if !ok {
			return fmt.Errorf("failed to get workflow by key after locking")
		}
		un, ok = obj.(*unstructured.Unstructured)
		if !ok {
			return fmt.Errorf("object %+v is not an unstructured", obj)
		}
		wf, err = util.FromUnstructured(un)
		if err != nil {
			return err
		}

		// workflow might still be hydrated
		if wfc.hydrator.IsHydrated(wf) {
			logger.WithField("uid", wf.UID).Info(ctx, "Hydrated workflow encountered")
			err = wfc.hydrator.Dehydrate(ctx, wf)
			if err != nil {
				return err
			}
		}

View on GitHub (pinned to 35bff19146)

Solutions

1. Treat as transient: the GC loop logs it and retries next period once the deletion settles.
2. Verify the workflow was actually deleted (kubectl get wf) — if so, the offload versions will be cleaned on the next pass.
3. If it recurs persistently for a live workflow, check informer health / API-server connectivity and cache sync state.

// Check existence under the same lock before proceeding
key := ns + "/" + name
wfc.workflowKeyLock.Lock(key)
_, ok := wfc.getWorkflowByKey(ctx, key)
wfc.workflowKeyLock.Unlock(key)

obj, ok := wfc.getWorkflowByKey(ctx, key)
if !ok {
    // benign race: workflow deleted between index lookup and lock
    return nil // or skip deleting this workflow's offloads this cycle
}

When it happens

Trigger: The workflow object was deleted from the cluster (or evicted from the informer cache) between the ByIndex(UIDIndex) call and getWorkflowByKey under the workflowKeyLock during offloaded-node-status GC.

Common situations: Workflows being deleted while their offloaded node status is being garbage collected — a transient race during heavy workflow cleanup; GC running right after a user runs `argo delete`.

Related errors

• object %+v is not an unstructured
• expected no more than 1 workflow, got %d
• failed to get pod from informer store: %w
• failed to delete ConfigMap %s: %w
• failed to update ConfigMap %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/f24e572353445875. Report an issue: GitHub.