{"record":{"id":"3e68035eea80d0cc","repo":"argoproj/argo-workflows","slug":"failed-to-hydrate-workflow-w","errorCode":null,"errorMessage":"failed to hydrate workflow: %w","messagePattern":"failed to hydrate workflow: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/controller/controller.go","lineNumber":1401,"sourceCode":"\t})\n\tif err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\nfunc (wfc *WorkflowController) archiveWorkflowAux(ctx context.Context, obj any) error {\n\tun, ok := obj.(*unstructured.Unstructured)\n\tif !ok {\n\t\treturn nil\n\t}\n\twf, err := util.FromUnstructured(un)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to convert to workflow from unstructured: %w\", err)\n\t}\n\terr = wfc.hydrator.Hydrate(ctx, wf)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to hydrate workflow: %w\", err)\n\t}\n\tlogger := logging.RequireLoggerFromContext(ctx)\n\tlogger.WithFields(logging.Fields{\"namespace\": wf.Namespace, \"workflow\": wf.Name, \"uid\": wf.UID}).Info(ctx, \"archiving workflow\")\n\terr = wfc.wfArchive.ArchiveWorkflow(ctx, wf)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to archive workflow: %w\", err)\n\t}\n\tdata, err := json.Marshal(map[string]any{\n\t\t\"metadata\": metav1.ObjectMeta{\n\t\t\tLabels: map[string]string{\n\t\t\t\tcommon.LabelKeyWorkflowArchivingStatus: \"Archived\",\n\t\t\t},\n\t\t},\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to marshal patch: %w\", err)\n\t}\n\t_, err = wfc.wfclientset.ArgoprojV1alpha1().Workflows(un.GetNamespace()).Patch(","sourceCodeStart":1383,"sourceCodeEnd":1419,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/controller.go#L1383-L1419","documentation":"archiveWorkflowAux (workflow/controller/controller.go:1401) fails when wfc.hydrator.Hydrate cannot materialize the workflow's offloaded node status back into Status.Nodes before archiving. Workflows whose node status is too large are offloaded to the configured database; archiving needs them re-inlined, so a hydrate failure blocks the workflow from being archived.","triggerScenarios":"Hydrate fetches the offloaded node-status version from the offloadNodeStatusRepo (Postgres/MySQL archive DB) and it fails — the offload record is missing or unreadable, the archive DB is down/unreachable, or the offloadEnabled config disagrees with the DB that holds the offloads.","commonSituations":"Archive database (postgres/mysql) outage or wrong DSN; persistedoffload rows deleted by an external GC or manual cleanup; switching archive configuration so the controller looks in the wrong table/database; network issues between controller and DB.","solutions":["Check the archive DB (persistence config in workflow-controller-configmap) connectivity and credentials; the wrapped error names the cause.","Verify the offloaded row exists (workflow offloads/node_status_version table) for the workflow UID/version; restore from backup if deleted.","Ensure offloadNodeStatusRepo config matches the DB that originally stored the offloads (don't switch persistence backends without migrating data).","Restart the controller once the DB is healthy — archiving is retried for workflows pending archive."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Check archive DB reachability before controller start\nimport _ \"github.com/lib/pq\"\ndb, err := sql.Open(\"postgres\", dsn)\nif err != nil || db.Ping() != nil { log.Fatal(\"archive DB unreachable\") }","typeGuard":null,"tryCatchPattern":"err := wfc.hydrator.Hydrate(ctx, wf)\nif err != nil {\n    if isTransientDBErr(err) { requeueWithBackoff(err) } // DB may recover\n    return fmt.Errorf(\"failed to hydrate workflow: %w\", err)\n}","preventionTips":["Monitor archive DB connectivity and credentials from the controller","Never delete offload rows manually or switch persistence backends without migrating data","Alert on 'failed to hydrate workflow' controller log lines","Keep retention/GC of offloaded rows under controller control only"],"tags":["database","hydration","archiving","postgres","mysql"],"backgroundTag":"hydration-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}