argoproj/argo-workflows · critical

reconnection failed after %d retries, last error: %w

Error message

reconnection failed after %d retries, last error: %w

What it means

reconnectLocked exhausted all retry attempts (maxRetries with linear backoff, bounded by maxDelay) trying to re-establish the database session and every attempt failed; err wraps the last connection error. Callers (reconnectIfStale within With, or explicit Reconnect) surface this wrapped cause.

Source

Thrown at util/sqldb/session.go:365

		// If this is the last attempt, don't wait
		if attempt == sp.maxRetries || !sp.isNetworkError(err) {
			break
		}

		// Calculate delay for next retry with linear backoff
		delay := time.Duration(float64(sp.baseDelay) * float64(attempt+1) * sp.retryMultiple)
		delay = min(delay, sp.maxDelay)

		// Wait before retrying with context cancellation support
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-time.After(delay):
		}
	}

	return fmt.Errorf("reconnection failed after %d retries, last error: %w", sp.maxRetries, err)
}

// Session returns the underlying session. Use With() for operations that need reconnection.
// This method is provided for cases where you need direct access to the session,
// but it won't provide automatic reconnection.
func (sp *SessionProxy) Session() db.Session {
	sp.mu.RLock()
	defer sp.mu.RUnlock()
	return sp.sess
}

// Close closes the session proxy and underlying session
func (sp *SessionProxy) Close() error {
	sp.mu.Lock()
	defer sp.mu.Unlock()

	if sp.closed {
		return nil

View on GitHub (pinned to 35bff19146)

Solutions

1. Unwrap the last error (%w) to see why connect failed and fix that root cause (network, DNS, auth, DB health).
2. Confirm DB credentials/username+password secrets are current after rotations.
3. Increase maxRetries / baseDelay / maxDelay in SessionProxy configuration for longer outages.
4. Ensure the calling context is not cancelled/expiring before the backoff loop finishes.
5. Restore the database itself (check StatefulSet/pod status, DB logs, max_connections).

Example fix

// before
proxy := NewSessionProxy(connect, 2, 500*time.Millisecond, 1.0, 2*time.Second) // fails on long outages
// after
proxy := NewSessionProxy(connect, 10, 500*time.Millisecond, 1.0, 30*time.Second) // ~ ride out restarts

// preflight connectivity check before issuing operations
conn, err := net.DialTimeout("tcp", dbHostPort, 3*time.Second)
if err != nil {
	return fmt.Errorf("database unreachable before operation: %w", err)
}
conn.Close()

if err := proxy.Reconnect(ctx); err != nil {
	if strings.Contains(err.Error(), "reconnection failed after") {
		// retry budget exhausted; escalate / circuit-break
		logger.Error(ctx, "db unavailable after retries", err)
	}
}

When it happens

Trigger: connect(ctx) fails maxRetries+1 consecutive times — DB down, unreachable, or rejecting credentials — during automatic reconnection after a network error, or during an explicit Reconnect call.

Common situations: Extended database outage (pod crash-looping, node down); misrotated DB credentials in k8s secrets; NetworkPolicy/security-group blocking the controller; context deadline exceeded during backoff; non-network errors breaking out of the retry loop on the first attempt.

Related errors

• operation failed and reconnection failed: %w
• no active session after reconnection
• operation failed after reconnection: %w
• failed to stream artifact: %v
• CodeNotFound

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/d3fc48a071fc2a91. Report an issue: GitHub.