{"record":{"id":"f24e572353445875","repo":"argoproj/argo-workflows","slug":"failed-to-get-workflow-by-key-after-locking","errorCode":null,"errorMessage":"failed to get workflow by key after locking","messagePattern":"failed to get workflow by key after locking","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"workflow/controller/controller.go","lineNumber":849,"sourceCode":"\t\treturn err\n\t}\n\tvar wf *wfv1.Workflow\n\tlogger := logging.RequireLoggerFromContext(ctx)\n\tswitch l := len(workflows); l {\n\tcase 0:\n\t\tlogger.WithField(\"uid\", uid).Info(ctx, \"Workflow missing, probably deleted\")\n\tcase 1:\n\t\tun, ok := workflows[0].(*unstructured.Unstructured)\n\t\tif !ok {\n\t\t\treturn fmt.Errorf(\"object %+v is not an unstructured\", workflows[0])\n\t\t}\n\t\tkey := un.GetNamespace() + \"/\" + un.GetName()\n\t\twfc.workflowKeyLock.Lock(key)\n\t\tdefer wfc.workflowKeyLock.Unlock(key)\n\n\t\tobj, ok := wfc.getWorkflowByKey(ctx, key)\n\t\tif !ok {\n\t\t\treturn fmt.Errorf(\"failed to get workflow by key after locking\")\n\t\t}\n\t\tun, ok = obj.(*unstructured.Unstructured)\n\t\tif !ok {\n\t\t\treturn fmt.Errorf(\"object %+v is not an unstructured\", obj)\n\t\t}\n\t\twf, err = util.FromUnstructured(un)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\t// workflow might still be hydrated\n\t\tif wfc.hydrator.IsHydrated(wf) {\n\t\t\tlogger.WithField(\"uid\", wf.UID).Info(ctx, \"Hydrated workflow encountered\")\n\t\t\terr = wfc.hydrator.Dehydrate(ctx, wf)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}","sourceCodeStart":831,"sourceCodeEnd":867,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/controller.go#L831-L867","documentation":"deleteOffloadedNodesForWorkflow (workflow/controller/controller.go:849) locks the workflow key, then calls getWorkflowByKey; if the key is no longer present in the informer cache the error is returned. This is a benign race: between the initial UID-index lookup and acquiring the per-key lock, the workflow was deleted (or the cache re-synced), so the object vanished.","triggerScenarios":"The workflow object was deleted from the cluster (or evicted from the informer cache) between the ByIndex(UIDIndex) call and getWorkflowByKey under the workflowKeyLock during offloaded-node-status GC.","commonSituations":"Workflows being deleted while their offloaded node status is being garbage collected — a transient race during heavy workflow cleanup; GC running right after a user runs `argo delete`.","solutions":["Treat as transient: the GC loop logs it and retries next period once the deletion settles.","Verify the workflow was actually deleted (kubectl get wf) — if so, the offload versions will be cleaned on the next pass.","If it recurs persistently for a live workflow, check informer health / API-server connectivity and cache sync state."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Check existence under the same lock before proceeding\nkey := ns + \"/\" + name\nwfc.workflowKeyLock.Lock(key)\n_, ok := wfc.getWorkflowByKey(ctx, key)\nwfc.workflowKeyLock.Unlock(key)","typeGuard":null,"tryCatchPattern":"obj, ok := wfc.getWorkflowByKey(ctx, key)\nif !ok {\n    // benign race: workflow deleted between index lookup and lock\n    return nil // or skip deleting this workflow's offloads this cycle\n}","preventionTips":["Treat absence-after-lock as a benign deletion race, not a hard failure","Let the periodic GC retry; offloads are cleaned once state settles","Monitor informer cache health if this error appears for live workflows"],"tags":["kubernetes","race-condition","informer","gc"],"backgroundTag":"workflow-deleted-race","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}