{"record":{"id":"f76c3f864230a48e","repo":"argoproj/argo-workflows","slug":"operation-failed-and-reconnection-failed-w","errorCode":null,"errorMessage":"operation failed and reconnection failed: %w","messagePattern":"operation failed and reconnection failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"util/sqldb/session.go","lineNumber":291,"sourceCode":"\tsess := sp.sess\n\tsp.mu.RUnlock()\n\n\tif sess == nil {\n\t\treturn fmt.Errorf(\"no active session\")\n\t}\n\n\terr := fn(sess)\n\tif err == nil {\n\t\treturn nil\n\t}\n\n\t// If it's not a network error or inside a tx do not retry\n\tif !sp.isNetworkError(err) || sp.insideTransaction {\n\t\treturn err\n\t}\n\n\tif reconnectErr := sp.reconnectIfStale(ctx, sess); reconnectErr != nil {\n\t\treturn fmt.Errorf(\"operation failed and reconnection failed: %w\", reconnectErr)\n\t}\n\n\tsp.mu.RLock()\n\tsess = sp.sess\n\tsp.mu.RUnlock()\n\n\tif sess == nil {\n\t\treturn fmt.Errorf(\"no active session after reconnection\")\n\t}\n\n\tif retryErr := fn(sess); retryErr != nil {\n\t\treturn fmt.Errorf(\"operation failed after reconnection: %w\", retryErr)\n\t}\n\n\treturn nil\n}\n\n// reconnectIfStale reconnects only if the current session is still the","sourceCodeStart":273,"sourceCodeEnd":309,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/util/sqldb/session.go#L273-L309","documentation":"The original operation failed with a network-class error, SessionProxy attempted an automatic reconnection via reconnectIfStale, and that reconnection itself failed. The returned error wraps the reconnection failure (which itself wraps the last connect error), so the root cause is the wrapped database connectivity error, not the original query.","triggerScenarios":"A query through With fails with a network error (driver.ErrBadConn, 'connection refused', 'server closed the connection', timeout, etc.) while sp.insideTransaction is false, and reconnectIfStale/reconnectLocked cannot establish a new session within maxRetries attempts with linear backoff.","commonSituations":"Database pod restart or network partition in the cluster; DB host unreachable (DNS/firewall); DB rejecting connections due to max_connections exhausted; context cancelled during reconnection backoff.","solutions":["Read the wrapped %w cause to identify the connectivity problem (refused vs timeout vs auth).","Verify network reachability from the controller pod to the DB (svc/DNS, NetworkPolicy, firewall, security groups).","Check the DB is accepting connections (pod running, max_connections not exhausted) and credentials/secrets are valid.","Tune SessionProxy retry parameters (maxRetries, baseDelay, maxDelay) to ride out brief outages, or fix the context deadline that is cancelling reconnection."],"exampleFix":"// before\nerr := proxy.With(ctx, fn) // \"operation failed and reconnection failed: ... dial tcp 10.0.0.5:5432: connect: connection refused\"\n// after\nif err != nil && strings.Contains(err.Error(), \"reconnection failed\") {\n\tif rerr := proxy.Reconnect(ctx); rerr != nil {\n\t\tlogger.Errorf(ctx, \"manual reconnect failed: %v\", rerr)\n\t}\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"var reconnErr *fmt.Errorf\nerr := proxy.With(ctx, fn)\nif err != nil && strings.Contains(err.Error(), \"operation failed and reconnection failed\") {\n\t// DB unreachable beyond retry budget; back off and alert\n\tlogger.Error(ctx, \"db reconnect failed during operation\", err)\n\treturn retryLater(ctx, fn)\n}","preventionTips":["Keep retry parameters (maxRetries, baseDelay, maxDelay) sized for realistic DB restart times.","Ensure the DB is deployed with a stable Service/DNS and health-checked before traffic.","Monitor network policies/firewalls between controller and DB.","Check context deadlines are generous enough to allow the reconnect backoff to complete."],"tags":["database","reconnection","network"],"backgroundTag":"db-reconnect-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}