{"record":{"id":"97fb5a9e4f4dc400","repo":"argoproj/argo-workflows","slug":"stried-to-offload-but-encountered-error-s","errorCode":null,"errorMessage":"%sTried to offload but encountered error: %s","messagePattern":"(.+?)Tried to offload but encountered error: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/hydrator/hydrator.go","lineNumber":120,"sourceCode":"\t\terr = packer.CompressWorkflowIfNeeded(ctx, wf)\n\t\tif err == nil {\n\t\t\twf.Status.OffloadNodeStatusVersion = \"\"\n\t\t\treturn nil\n\t\t}\n\t}\n\tif packer.IsTooLargeError(err) || h.alwaysOffloadNodeStatus {\n\t\tvar offloadVersion string\n\t\tvar errMsg string\n\t\tif err != nil {\n\t\t\terrMsg += err.Error()\n\t\t}\n\t\toffloadErr := waitutil.Backoff(writeRetry, func() (bool, error) {\n\t\t\tvar offloadErr error\n\t\t\toffloadVersion, offloadErr = h.offloadNodeStatusRepo.Save(ctx, string(wf.UID), wf.Namespace, wf.Status.Nodes)\n\t\t\treturn !errorsutil.IsTransientErr(ctx, offloadErr), offloadErr\n\t\t})\n\t\tif offloadErr != nil {\n\t\t\treturn fmt.Errorf(\"%sTried to offload but encountered error: %s\", errMsg, offloadErr.Error())\n\t\t}\n\t\twf.Status.Nodes = nil\n\t\twf.Status.CompressedNodes = \"\"\n\t\twf.Status.OffloadNodeStatusVersion = offloadVersion\n\t\treturn nil\n\t}\n\treturn err\n}\n","sourceCodeStart":102,"sourceCodeEnd":129,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/hydrator/hydrator.go#L102-L129","documentation":"Dehydrate() moves a Workflow's large Status.Nodes off the object itself into the offload repository (SQL) when the node status is too big to store in the CRD. If the offload save still fails after retrying transient errors with waitutil.Backoff, this error wraps the final failure — the workflow cannot be updated with reduced size and the controller will retry the whole operation.","triggerScenarios":"h.offloadNodeStatusRepo.Save repeatedly returns a non-transient error (or exhausts the writeRetry backoff) while dehydrating an oversized workflow — typically during the controller's operational update of a workflow with thousands of nodes.","commonSituations":"Database down or connection pool exhausted; archive/offload tables missing or migrated incorrectly (older workflow-controller upgrading without running the DB migration); DB credentials/permission errors; context deadline exceeded because the backoff outlived the reconcile timeout with a huge node set.","solutions":["Check the offload database health and connectivity (Postgres/MySQL used by persist/sqldb) and the wrapped offloadErr message","Verify the workflow and node_status_offload schema/tables exist and migrations ran (the controller logs migration status on startup)","Confirm ConfigMap persistence config (persistence.enabled, nodeStatusOffload: true) matches the deployed DB","Retry the controller / wait for waitutil.Backoff — transient DB blips self-heal; persistent errors need the DB fixed","If a single workflow is pathologically large, consider splitting the workflow (nested/dynamic fan-out) rather than growing node count"],"exampleFix":"// before: persistence misconfigured\nnodeStatusOffload: false  # large workflows fail to dehydrate\n// after\nnodeStatusOffload: true\narchive: true\npostgresql:\n  host: postgres\n  database: postgres\n","handlingStrategy":"retry","validationCode":"// before submitting large workflows, confirm persistence is configured\n// kubectl get cm workflow-controller-configmap -o yaml | grep -A5 persistence","typeGuard":null,"tryCatchPattern":"err := hydrator.Dehydrate(ctx, wf)\nif err != nil {\n    if offloadErr := strings.Contains(err.Error(), \"Tried to offload\"); offloadErr {\n        // check DB connectivity/schema, then requeue for another reconcile attempt\n    }\n}","preventionTips":["Enable nodeStatusOffload in the persistence config when workflows exceed CRD size limits","Run DB migrations when upgrading the controller before processing large archived workflows","Monitor offload DB health (connections, disk) — it is on the controller hot path","Split pathologically large workflows (nested templates/dynamic fan-out) to keep node counts manageable"],"tags":["database","offload","hydrator","controller","workflow"],"backgroundTag":"node-status-offload-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}