What causes "session proxy is closed" in argoproj/argo-workflows?

Do not call persistence APIs after the proxy is closed; gate callers on application shutdown ordering (stop consumers before closing the DB session).

argoproj/argo-workflows · error

session proxy is closed

Error message

session proxy is closed

What it means

SessionProxy.With refuses to run the caller's function because the proxy has been marked closed (sp.closed=true). The proxy is closed either explicitly via Close() or transiently inside reconnectLocked, which sets closed=true before reconnecting. Once closed, all persistence operations routed through With (list, get, save, offload cleanup) fail immediately.

Source

Thrown at util/sqldb/session.go:270

		return true
	}

	// Check for sql.ErrConnDone
	if errors.Is(err, sql.ErrConnDone) {
		return true
	}

	return false
}

// With executes with a db Session
func (sp *SessionProxy) With(ctx context.Context, fn func(db.Session) error) error {
	logger := logging.RequireLoggerFromContext(ctx)
	sp.mu.RLock()
	if sp.closed {
		sp.mu.RUnlock()
		logger.Warn(ctx, "session proxy is closed")
		return fmt.Errorf("session proxy is closed")
	}

	sess := sp.sess
	sp.mu.RUnlock()

	if sess == nil {
		return fmt.Errorf("no active session")
	}

	err := fn(sess)
	if err == nil {
		return nil
	}

	// If it's not a network error or inside a tx do not retry
	if !sp.isNetworkError(err) || sp.insideTransaction {
		return err
	}

View on GitHub (pinned to 35bff19146)

Solutions

Do not call persistence APIs after the proxy is closed; gate callers on application shutdown ordering (stop consumers before closing the DB session).
Check the underlying error: if this appeared after a network outage, restart/reconnect the persistence layer (create a new offload repo via NewPersistence) instead of reusing the closed proxy.
If it happens during reconnection retries, wait for the retry loop to finish; errors are expected transiently while reconnectLocked is running.
Increase maxRetries/baseDelay in the SessionProxy config so reconnection succeeds before callers give up.

Example fix

// before
repo.Close(ctx)
workflows, err := repo.ListWorkflows(ctx, options) // "session proxy is closed"
// after
if err := repo.Close(ctx); err != nil { ... }
// stop using repo afterwards; create a fresh one if needed
repo, err := sqldb.NewPersistence(sessionFactory, tableName, nil, instanceID, false, nil)

Defensive patterns

Strategy: try-catch

Try / catch

if err := repo.Save(ctx, wf); err != nil {
	if strings.Contains(err.Error(), "session proxy is closed") {
		logger.Warn(ctx, "persistence unavailable (closed); skipping save")
		return nil // graceful shutdown path
	}
	return err
}

Prevention

Close the session proxy only during final shutdown, after all worker goroutines have drained.
Use WaitGroups/sync to guarantee no in-flight persistence calls when Close() runs.
Treat 'closed' errors as a signal to create a fresh persistence instance rather than reuse the old proxy.
Log the shutdown order so closed-session errors are traceable.

When it happens

Trigger: Calling any persistence-backed API (ListWorkflowsLabelKeys, ListWorkflowsLabelValues, Save, Get, List, ListOldOffloads) after SessionProxy.Close() was called, or concurrently while reconnectLocked is mid-retry (it closes the session and sets closed=true between attempts).

Common situations: Controller shutdown racing with in-flight archive/offload queries; a failed reconnection leaving the proxy permanently closed; manual Close() then reuse of a persisted offload repo; tests tearing down the DB session while background goroutines still poll.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/56ec6a8b0d976451. Report an issue: GitHub.