What causes "failed to hydrate workflow: %w" in argoproj/argo-workflows?

Check the archive DB (persistence config in workflow-controller-configmap) connectivity and credentials; the wrapped error names the cause.

argoproj/argo-workflows · error

failed to hydrate workflow: %w

Error message

failed to hydrate workflow: %w

What it means

archiveWorkflowAux (workflow/controller/controller.go:1401) fails when wfc.hydrator.Hydrate cannot materialize the workflow's offloaded node status back into Status.Nodes before archiving. Workflows whose node status is too large are offloaded to the configured database; archiving needs them re-inlined, so a hydrate failure blocks the workflow from being archived.

Source

Thrown at workflow/controller/controller.go:1401

	})
	if err != nil {
		return err
	}
	return nil
}

func (wfc *WorkflowController) archiveWorkflowAux(ctx context.Context, obj any) error {
	un, ok := obj.(*unstructured.Unstructured)
	if !ok {
		return nil
	}
	wf, err := util.FromUnstructured(un)
	if err != nil {
		return fmt.Errorf("failed to convert to workflow from unstructured: %w", err)
	}
	err = wfc.hydrator.Hydrate(ctx, wf)
	if err != nil {
		return fmt.Errorf("failed to hydrate workflow: %w", err)
	}
	logger := logging.RequireLoggerFromContext(ctx)
	logger.WithFields(logging.Fields{"namespace": wf.Namespace, "workflow": wf.Name, "uid": wf.UID}).Info(ctx, "archiving workflow")
	err = wfc.wfArchive.ArchiveWorkflow(ctx, wf)
	if err != nil {
		return fmt.Errorf("failed to archive workflow: %w", err)
	}
	data, err := json.Marshal(map[string]any{
		"metadata": metav1.ObjectMeta{
			Labels: map[string]string{
				common.LabelKeyWorkflowArchivingStatus: "Archived",
			},
		},
	})
	if err != nil {
		return fmt.Errorf("failed to marshal patch: %w", err)
	}
	_, err = wfc.wfclientset.ArgoprojV1alpha1().Workflows(un.GetNamespace()).Patch(

View on GitHub (pinned to 35bff19146)

Solutions

Check the archive DB (persistence config in workflow-controller-configmap) connectivity and credentials; the wrapped error names the cause.
Verify the offloaded row exists (workflow offloads/node_status_version table) for the workflow UID/version; restore from backup if deleted.
Ensure offloadNodeStatusRepo config matches the DB that originally stored the offloads (don't switch persistence backends without migrating data).
Restart the controller once the DB is healthy — archiving is retried for workflows pending archive.

Defensive patterns

Strategy: retry

Validate before calling

// Check archive DB reachability before controller start
import _ "github.com/lib/pq"
db, err := sql.Open("postgres", dsn)
if err != nil || db.Ping() != nil { log.Fatal("archive DB unreachable") }

Try / catch

err := wfc.hydrator.Hydrate(ctx, wf)
if err != nil {
    if isTransientDBErr(err) { requeueWithBackoff(err) } // DB may recover
    return fmt.Errorf("failed to hydrate workflow: %w", err)
}

Prevention

Monitor archive DB connectivity and credentials from the controller
Never delete offload rows manually or switch persistence backends without migrating data
Alert on 'failed to hydrate workflow' controller log lines
Keep retention/GC of offloaded rows under controller control only

When it happens

Trigger: Hydrate fetches the offloaded node-status version from the offloadNodeStatusRepo (Postgres/MySQL archive DB) and it fails — the offload record is missing or unreadable, the archive DB is down/unreachable, or the offloadEnabled config disagrees with the DB that holds the offloads.

Common situations: Archive database (postgres/mysql) outage or wrong DSN; persistedoffload rows deleted by an external GC or manual cleanup; switching archive configuration so the controller looks in the wrong table/database; network issues between controller and DB.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/3e68035eea80d0cc. Report an issue: GitHub.