{"record":{"id":"56ec6a8b0d976451","repo":"argoproj/argo-workflows","slug":"session-proxy-is-closed","errorCode":null,"errorMessage":"session proxy is closed","messagePattern":"session proxy is closed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"util/sqldb/session.go","lineNumber":270,"sourceCode":"\t\treturn true\n\t}\n\n\t// Check for sql.ErrConnDone\n\tif errors.Is(err, sql.ErrConnDone) {\n\t\treturn true\n\t}\n\n\treturn false\n}\n\n// With executes with a db Session\nfunc (sp *SessionProxy) With(ctx context.Context, fn func(db.Session) error) error {\n\tlogger := logging.RequireLoggerFromContext(ctx)\n\tsp.mu.RLock()\n\tif sp.closed {\n\t\tsp.mu.RUnlock()\n\t\tlogger.Warn(ctx, \"session proxy is closed\")\n\t\treturn fmt.Errorf(\"session proxy is closed\")\n\t}\n\n\tsess := sp.sess\n\tsp.mu.RUnlock()\n\n\tif sess == nil {\n\t\treturn fmt.Errorf(\"no active session\")\n\t}\n\n\terr := fn(sess)\n\tif err == nil {\n\t\treturn nil\n\t}\n\n\t// If it's not a network error or inside a tx do not retry\n\tif !sp.isNetworkError(err) || sp.insideTransaction {\n\t\treturn err\n\t}","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/util/sqldb/session.go#L252-L288","documentation":"SessionProxy.With refuses to run the caller's function because the proxy has been marked closed (sp.closed=true). The proxy is closed either explicitly via Close() or transiently inside reconnectLocked, which sets closed=true before reconnecting. Once closed, all persistence operations routed through With (list, get, save, offload cleanup) fail immediately.","triggerScenarios":"Calling any persistence-backed API (ListWorkflowsLabelKeys, ListWorkflowsLabelValues, Save, Get, List, ListOldOffloads) after SessionProxy.Close() was called, or concurrently while reconnectLocked is mid-retry (it closes the session and sets closed=true between attempts).","commonSituations":"Controller shutdown racing with in-flight archive/offload queries; a failed reconnection leaving the proxy permanently closed; manual Close() then reuse of a persisted offload repo; tests tearing down the DB session while background goroutines still poll.","solutions":["Do not call persistence APIs after the proxy is closed; gate callers on application shutdown ordering (stop consumers before closing the DB session).","Check the underlying error: if this appeared after a network outage, restart/reconnect the persistence layer (create a new offload repo via NewPersistence) instead of reusing the closed proxy.","If it happens during reconnection retries, wait for the retry loop to finish; errors are expected transiently while reconnectLocked is running.","Increase maxRetries/baseDelay in the SessionProxy config so reconnection succeeds before callers give up."],"exampleFix":"// before\nrepo.Close(ctx)\nworkflows, err := repo.ListWorkflows(ctx, options) // \"session proxy is closed\"\n// after\nif err := repo.Close(ctx); err != nil { ... }\n// stop using repo afterwards; create a fresh one if needed\nrepo, err := sqldb.NewPersistence(sessionFactory, tableName, nil, instanceID, false, nil)","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := repo.Save(ctx, wf); err != nil {\n\tif strings.Contains(err.Error(), \"session proxy is closed\") {\n\t\tlogger.Warn(ctx, \"persistence unavailable (closed); skipping save\")\n\t\treturn nil // graceful shutdown path\n\t}\n\treturn err\n}","preventionTips":["Close the session proxy only during final shutdown, after all worker goroutines have drained.","Use WaitGroups/sync to guarantee no in-flight persistence calls when Close() runs.","Treat 'closed' errors as a signal to create a fresh persistence instance rather than reuse the old proxy.","Log the shutdown order so closed-session errors are traceable."],"tags":["database","session-lifecycle","concurrency"],"backgroundTag":"connection-pool-closed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}