argoproj/argo-workflows · error

%sTried to offload but encountered error: %s

Error message

%sTried to offload but encountered error: %s

What it means

Dehydrate() moves a Workflow's large Status.Nodes off the object itself into the offload repository (SQL) when the node status is too big to store in the CRD. If the offload save still fails after retrying transient errors with waitutil.Backoff, this error wraps the final failure — the workflow cannot be updated with reduced size and the controller will retry the whole operation.

Source

Thrown at workflow/hydrator/hydrator.go:120

		err = packer.CompressWorkflowIfNeeded(ctx, wf)
		if err == nil {
			wf.Status.OffloadNodeStatusVersion = ""
			return nil
		}
	}
	if packer.IsTooLargeError(err) || h.alwaysOffloadNodeStatus {
		var offloadVersion string
		var errMsg string
		if err != nil {
			errMsg += err.Error()
		}
		offloadErr := waitutil.Backoff(writeRetry, func() (bool, error) {
			var offloadErr error
			offloadVersion, offloadErr = h.offloadNodeStatusRepo.Save(ctx, string(wf.UID), wf.Namespace, wf.Status.Nodes)
			return !errorsutil.IsTransientErr(ctx, offloadErr), offloadErr
		})
		if offloadErr != nil {
			return fmt.Errorf("%sTried to offload but encountered error: %s", errMsg, offloadErr.Error())
		}
		wf.Status.Nodes = nil
		wf.Status.CompressedNodes = ""
		wf.Status.OffloadNodeStatusVersion = offloadVersion
		return nil
	}
	return err
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the offload database health and connectivity (Postgres/MySQL used by persist/sqldb) and the wrapped offloadErr message
2. Verify the workflow and node_status_offload schema/tables exist and migrations ran (the controller logs migration status on startup)
3. Confirm ConfigMap persistence config (persistence.enabled, nodeStatusOffload: true) matches the deployed DB
4. Retry the controller / wait for waitutil.Backoff — transient DB blips self-heal; persistent errors need the DB fixed
5. If a single workflow is pathologically large, consider splitting the workflow (nested/dynamic fan-out) rather than growing node count

Example fix

// before: persistence misconfigured
nodeStatusOffload: false  # large workflows fail to dehydrate
// after
nodeStatusOffload: true
archive: true
postgresql:
  host: postgres
  database: postgres

// before submitting large workflows, confirm persistence is configured
// kubectl get cm workflow-controller-configmap -o yaml | grep -A5 persistence

err := hydrator.Dehydrate(ctx, wf)
if err != nil {
    if offloadErr := strings.Contains(err.Error(), "Tried to offload"); offloadErr {
        // check DB connectivity/schema, then requeue for another reconcile attempt
    }
}

When it happens

Trigger: h.offloadNodeStatusRepo.Save repeatedly returns a non-transient error (or exhausts the writeRetry backoff) while dehydrating an oversized workflow — typically during the controller's operational update of a workflow with thousands of nodes.

Common situations: Database down or connection pool exhausted; archive/offload tables missing or migrated incorrectly (older workflow-controller upgrading without running the DB migration); DB credentials/permission errors; context deadline exceeded because the backoff outlived the reconcile timeout with a huge node set.

Related errors

• offload node status is not supported
• failed to parse Workflow: %w
• artifact driver %s not found
• if you have an item in your config map named 'config', you m
• operation %v is not supported

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/97fb5a9e4f4dc400. Report an issue: GitHub.