argoproj/argo-workflows · error

operation failed after reconnection: %w

Error message

operation failed after reconnection: %w

What it means

SessionProxy reconnected successfully after a network error and retried the caller's function on the new session, but the retry also failed. The retry error is wrapped with this prefix; the original error was discarded in favor of the more relevant post-reconnect failure.

Source

Thrown at util/sqldb/session.go:303

	// If it's not a network error or inside a tx do not retry
	if !sp.isNetworkError(err) || sp.insideTransaction {
		return err
	}

	if reconnectErr := sp.reconnectIfStale(ctx, sess); reconnectErr != nil {
		return fmt.Errorf("operation failed and reconnection failed: %w", reconnectErr)
	}

	sp.mu.RLock()
	sess = sp.sess
	sp.mu.RUnlock()

	if sess == nil {
		return fmt.Errorf("no active session after reconnection")
	}

	if retryErr := fn(sess); retryErr != nil {
		return fmt.Errorf("operation failed after reconnection: %w", retryErr)
	}

	return nil
}

// reconnectIfStale reconnects only if the current session is still the
// same one that produced the error. If another goroutine already
// reconnected (sp.sess != staleSess), this is a no-op.
func (sp *SessionProxy) reconnectIfStale(ctx context.Context, staleSess db.Session) error {
	sp.mu.Lock()
	defer sp.mu.Unlock()
	if sp.sess != staleSess {
		return nil
	}
	return sp.reconnectLocked(ctx)
}

// Reconnect performs reconnection with retry logic and linear backoff

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the wrapped cause of the retry error — it reflects the real problem now that connectivity is restored.
2. Make operations idempotent (upserts) so a retry after reconnect is safe.
3. Check DB-side state: permissions, schema, and whether the retry hit a context cancellation/deadline.
4. If the retry always fails while the first attempt succeeded historically, compare connection config (DB name, user, TLS) between old and new sessions.

Example fix

// before
err := proxy.With(ctx, save) // "operation failed after reconnection: pq: duplicate key value violates unique constraint"
// after
// make save idempotent
sess.Insert(orUpdate(wf)) // INSERT ... ON CONFLICT (name) DO UPDATE

err := proxy.With(ctx, fn)
if err != nil {
	var postReconn bool
	if strings.Contains(err.Error(), "operation failed after reconnection") {
		postReconn = true
	}
	if postReconn {
		// connectivity is fine; the failure is application/DB-level — do not blind-retry
		return classifyAndHandle(err)
	}
}

When it happens

Trigger: fn(sess) fails a second time on the fresh session after automatic reconnection — i.e. the failure is not purely transient connectivity (e.g. query error, constraint violation, permission error, or DB that reconnects at TCP level but rejects queries).

Common situations: DB restored but schema/permissions changed; the operation is non-idempotent and partially applied before the network blip; the new session lands on a replica or different DB with different state; context deadline expires during retry.

Related errors

• operation failed and reconnection failed: %w
• no active session after reconnection
• reconnection failed after %d retries, last error: %w
• requires either node field selector or workflow
• resolve UID: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/967b75c29d661e6b. Report an issue: GitHub.