argoproj/argo-workflows · critical

operation failed and reconnection failed: %w

Error message

operation failed and reconnection failed: %w

What it means

The original operation failed with a network-class error, SessionProxy attempted an automatic reconnection via reconnectIfStale, and that reconnection itself failed. The returned error wraps the reconnection failure (which itself wraps the last connect error), so the root cause is the wrapped database connectivity error, not the original query.

Source

Thrown at util/sqldb/session.go:291

	sess := sp.sess
	sp.mu.RUnlock()

	if sess == nil {
		return fmt.Errorf("no active session")
	}

	err := fn(sess)
	if err == nil {
		return nil
	}

	// If it's not a network error or inside a tx do not retry
	if !sp.isNetworkError(err) || sp.insideTransaction {
		return err
	}

	if reconnectErr := sp.reconnectIfStale(ctx, sess); reconnectErr != nil {
		return fmt.Errorf("operation failed and reconnection failed: %w", reconnectErr)
	}

	sp.mu.RLock()
	sess = sp.sess
	sp.mu.RUnlock()

	if sess == nil {
		return fmt.Errorf("no active session after reconnection")
	}

	if retryErr := fn(sess); retryErr != nil {
		return fmt.Errorf("operation failed after reconnection: %w", retryErr)
	}

	return nil
}

// reconnectIfStale reconnects only if the current session is still the

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped %w cause to identify the connectivity problem (refused vs timeout vs auth).
2. Verify network reachability from the controller pod to the DB (svc/DNS, NetworkPolicy, firewall, security groups).
3. Check the DB is accepting connections (pod running, max_connections not exhausted) and credentials/secrets are valid.
4. Tune SessionProxy retry parameters (maxRetries, baseDelay, maxDelay) to ride out brief outages, or fix the context deadline that is cancelling reconnection.

Example fix

// before
err := proxy.With(ctx, fn) // "operation failed and reconnection failed: ... dial tcp 10.0.0.5:5432: connect: connection refused"
// after
if err != nil && strings.Contains(err.Error(), "reconnection failed") {
	if rerr := proxy.Reconnect(ctx); rerr != nil {
		logger.Errorf(ctx, "manual reconnect failed: %v", rerr)
	}
}

var reconnErr *fmt.Errorf
err := proxy.With(ctx, fn)
if err != nil && strings.Contains(err.Error(), "operation failed and reconnection failed") {
	// DB unreachable beyond retry budget; back off and alert
	logger.Error(ctx, "db reconnect failed during operation", err)
	return retryLater(ctx, fn)
}

When it happens

Trigger: A query through With fails with a network error (driver.ErrBadConn, 'connection refused', 'server closed the connection', timeout, etc.) while sp.insideTransaction is false, and reconnectIfStale/reconnectLocked cannot establish a new session within maxRetries attempts with linear backoff.

Common situations: Database pod restart or network partition in the cluster; DB host unreachable (DNS/firewall); DB rejecting connections due to max_connections exhausted; context cancelled during reconnection backoff.

Related errors

• reconnection failed after %d retries, last error: %w
• no active session after reconnection
• operation failed after reconnection: %w
• failed to stream artifact: %v
• CodeNotFound

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/f76c3f864230a48e. Report an issue: GitHub.